CODE

The latest version of the code is on the GitHub: https://github.com/booydar/LM-RMT

Language model experiments and algorithmic (copy, reverse, associative retrieval) code is located in ./transformer-xl-memory_experiments
Code for experiments with quadratic equations is in ./transformer-xl-math.

Our code is forked from the Transformer-XL repository https://github.com/kimiyoung/transformer-xl.
We made changes to the Transformer-XL model in PyTorch scripts to implement Recurrent Memory Transformer.

All experiments results were produced with this repo and we will be publicly available.

Raw experiments results:

Results for all experiments are in results_csv folder:
    - short_synthetic+MT.csv -- experiments with copy/reverse/associative retrieval from figure 4.
    - long_copy_reverse.csv -- results for sequence length up to 1080 for copy/revers, figure 5a, 5b and 7c.
    - results_wt103_enwik8_all_parameters.csv -- full set of hyperparameters for enwik8/wt103 runs.
    - results_wt103_enwik8.csv -- enwik8/wt103 with selected subset of hyperparameters


Data.

LM on WT103 and enwik8:
    - following instructions from the Transformer-XL repository.

Copy & Reverse data:
    - data_generation/algorithmic.ipynb
    
Associative retrieval data:
    - data generation code: https://github.com/GokuMohandas/fast-weights/blob/539fb10e3c384d5f782af2560bf28631cd0eaa61/fw/data_utils.py

Quadratic equations data:
    - data_generation/square_equations.ipynb
