# Comparison with approximate inference

This folder contains the following contents:

1. `anglican/`: contains all the benchmarks translated to Anglican and the JSON results of Anglican's inference algorithms on them (in `anglican/output`).
2. `exact_output/`: contains the JSON results of exact inference on the benchmark (as described in `../genfer/README.md`).
3. `plots/`: the plots showing the results are saved here.
4. `analysis.ipynb`: the Python notebook to analyze the JSON results and to produce the plots.

## Reproducing the experimental results

To reproduce the experimental results, there are two steps: generating the data (i.e. running our tool and Anglican) and processing the data.
The data is already included in this artifact, so you can skip the first step if you only care about the data analysis.

### Step 1: generating the data

1. Build Genfer as described in `../genfer/README.md` and run `python3 generate_genfer_data.py`.
   The output files will be written to `exact_output/`.
2. Run the Anglican inference algorithms as described in `anglican/README.md`.
   The output files will be written to `anglican/output/`.

### Step 2: analyzing the data

The data was analyzed by running `analysis.ipynb` for each benchmark.
The plots are saved to `plots/` and are the ones used in the paper.

To reproduce this, we assume that you have a Python 3 and Jupyter installation on your system.
You need to install the packages: `glob`, `json5`, `numpy`, `scipy`, `pandas`, and `matplotlib`.
If you are missing any, you can easily install them with the command:

```
pip3 install glob json5 numpy scipy pandas matplotlib nbconvert
```

Then run `bash analyze_all.sh` to generate all the plots.
They will be saved to `plots/`.
