# SWE-agent

SWE-agent is an LM-based system for autonomously solving bug reports in software repositories. This repository provides an end-to-end solution, starting from a bug report to generating a pull request that fixes it. We use SWE-bench, a 2k problem set containing bugs from the top Python repositories on GitHub, to evaluate our model.

### Set Up
0. Install conda & docker, and then start docker. 
1. Create the `swe-agent` environment with `conda env create -f environment.yml`
2. Run `conda activate swe-agent`.
3. Run `./setup.sh` to create the `intercode-swe` docker image. Note: if you edit one of the files in the `docker/` folder you must run setup.sh again.
4. Create a `keys.cfg` file at the root of this repository and fill in the following:
```
OPENAI_API_KEY: 'OpenAI API Key Here'
ANTHROPIC_API_KEY: 'Anthropic API Key Here'
GITHUB_TOKEN: 'GitHub Token Here'
```

### Inference + Evaluation
1. Run models on SWEEnv and generate trajectories + patches:
```
python run.py --model_name gpt4 --per_instance_cost_limit 2.00 --config_file ./config/default.yaml
```

Other flags to be aware of:
- `--per_instance_cost_limit` specifies the cost limit for each task instance. Defaults to `2.00`.
- `--config_file` specifies the config to use for the agent, e.g., templates and commands. Defaults to `config/default.yaml`.
- `--temperature` specifies the temperature to use for sampling from the model. Defaults to `0.2`.
- `--top_p` specifies the top p value to use for sampling from the model. Defaults to `0.9`.
- `--install_environment` specifies whether to install the environment in the docker image. Defaults to `True`.
- `--container_name` specifies the name of the docker container to use. Creates a temporary container if not specified. By specifying the container name, you can avoid having to install the environment every time you run the script.
- `--instance_filter` can be set to a regex filter that matches instance id to run. Defaults to .*.
- Call `python run.py --help` for a full list!
 
    WARNING: If you specify a container name, do not run multiple instances of `run.py` with the same container name!

2. Evaluate patches with SWE-bench evaluation:
```
cd evaluation/
./run_eval.sh <predictions_path>
```
In `run_eval.sh`, make sure to replace `<predictions_path>` with the path to the model's generated patches from step 1.
The directory should look something like: `../trajectories/<username>/<model>-<dataset>-<hyperparams>/all_preds.jsonl`

*UNDER CONSTRUCTION
You can run an environment in your own repo-issue pair with:

`python run_single.py issue-file-1 issue-file-2 ... --model_name model --traj_dir path/to/trajectories`

- `issue-file` is a path to a YAML file containing the following fields:
```yaml
repo: owner/repo
query: |
  issue text
  more issue text. (make sure indentation is correct)
base_commit: 3e91bj0a # optional, defaults to HEAD
branch: my-branch # optional, defaults to default branch
```
See the file `data/custom/llama-issue.yaml` for an example of an issue file.

Note that if `branch` is specified, the `base_commit` must be left blank. You should not specify a branch to use if you are using a commit hash.*

For `run_single.py`, you can specify the following flags:
- `--template_file` to specify different templates for the environment. See the `templates` directory for examples.
- `--image_name` to specify a different docker image to use for the environment.

### Agent Config
The config files are used to specify different templates for prompting models. See the `AgentConfig` in `agent/agents.py`.
Configs are stored as `yaml` files in the `config` directory, e.g.,
```yaml
system_template: |
  SYSTEM...
instance_template: |-
  INSTANCE...
format_error_template: |-
  ERROR!!!
env_variables:
  WINDOW_SIZE=30
command_files:
  - path/to/bash_file.sh
state_command:
  name: state
  code: state() { echo '{"pwd": "'$PWD'"}';
```
where `SYSTEM`, `INSTANCE`, and `ERROR` are multi-line strings. The `|` character is used to denote a multi-line string in `yaml`, and `|-` is a multi-line string without trailing new-line. `command_files` parses a bash file of commands that will be parsed to the environment. `env_variables` are the default variables for the bash environment at the beginning of each instance. `state_command` is used to extract state information from the bash environment (formatted as json) to be used in the templates given to the agent.

Possible variables that can be used in templates are:
- `{command_docs}` (an automatically compiled collection of available commands + their docstrings)
- any variable given in `env_variables` (same spelling), e.g., `{WINDOW_SIZE}`
- any variable extracted as json as part of the `state_command` function
- the last observation `{observation}` 
- ... this list will grow as we implement more features!

### Environment Usage
* To skip over a task instance, use the `skip` keyword
* To submit for evaluation, use the `submit` keyword
* To exit the `SWEEnv` environment, perform a keyboard interrupt (`^ c`)
