# EVA

## Installation: 
From `eva/examples/NLU` install: 
```
conda env create -f environment.yml -n eva
conda activate eva
pip install -e .
```

Then from `eva` (base-dir) install `loralib`: 
```
python loralib/setup.py install
```
It is essentiall that no `build` dir or `loralib.eff-info` exists inside `eva`. Othewise installation fails. 


Install additional dependencies (move to `requirements.text` later): 
```
pip install matplotlib
pip install wandb
```

# Default Hyperparameters

| Task | n_gpus | batch size | epochs |
| -----|--------|------------|-----------|
| CoLA | 4 | 32         | 80 |
| STSB | 4 | 16         | 40 |
| MRPC | 4 | 16         | 30 |
| RTE | 4 | 32         | 80 |

# Gridsearches

| Parameter | Values                 |
|------|------------------------|
|r| 2,4,8,16               |
|lr| 4e-3, 1e-3, 4e-4, 1e-4 |


## Evaluation
This repository is using `wandb`. To generate results tables, we pull the metrics directly from `wandb`.

The main entry point for generating tables is `./postprocessing/plots_and_tables.py`. 
The script downloads the experiment with the specified `experiment_name` from `wandb` using `./postprocessing/wandb_utils.py`. 

Downloaded experiments are stored to the specified `cache_dir`.
By default, a file `[experiment_name].csv` is written to `./postprocessing/cache_dir`.
If new runs are saved to `wandb` with the same `experiment_name`, re-downloading can be forced using `--force_reload`.

After downloading the runs, tables can be generated. The experiment configuration are specified in `./postprocessing/exps.py`.
`plots_and_tables.py` looks for the specified `experiment_name`, extracts the experiment configuration and generates the table.

For example, to generate the table for the experiment `alpha_grid_pcainit` run: 
```
python plots_and_tables.py --api --table --exp_name=alpha_grid_pcainit
```

from within `./postprocessing`. By the default, tables are written to `./tables/[experiment_name]`, the location can be
re-configured using `--tab_dir`.
