## Setup
* python==3.8
* transformers>=3.3.0
* openai

## Data
Refer to the `data` folder in `Synth`, `HotpotAdv`, and `SNLI`.

## Experiments on Synth
Switch to `Synth` directory, run `sh main_exp.sh` (modify the `TR_SLICE` according to the annotations in the script).

## Experiments on HotpotAdv
Switch to `HotpotAdv`, run `python main_exp.py`

## Experiments on SNLI
Switch to `SNLI` directory, run `python main_exp.py`.

## Manually Annotated Explanations For Adversarial Hotpot Training Shots
See  `HotpotAdv/annotations/manual_rat_std.txt`.

## Reliability Annotations for Explanations of Adversarial Hotpot Examples
See `HotpotAdv/annotations/e-p_qualty_annotations.txt`.
