# Recurrent Random Networks as Optimized Kernel Machines

This repository is the official implementation of Recurrent Random Networks as Optimized Kernel Machines

## Requirements

To install, execute:

```setup
conda env create -f environment.yml
```

Additionally, the code uses the nest simulator [1]. It can be installed using

```install nest
conda install -c conda-forge nest-simulator
```

It is not included in the above installation file because of changes made by the authors to the nest source code, which are not yet merged to the official release. To generate the figures in the paper, most simulation data is included in the the `data/` folder.

[1] Gewaltig, Marc-Oliver and Diesmann, Markus. "Nest (neural simulation tool)." Scholarpedia 2.4 (2007): 1430. https://www.nest-simulator.org/.

## Structure

Please download[^1] the dataset `ECG5000`  from the UCR time series classification archive [2] and place the folder `ECG5000/`  next to `data/`. The folder `data/` contains all network configurations, figures and results written to files.

In all trainings and evaluations, the network realization has to be specified. These are stored under `data/network_realizations/`, listed by their seed number. A few predefined networks (seeds 1001 to 1010) are contained, but further can be created using

```setup
python -c 'import figures_2_3 as figs; figs.setup_network(seed)'
```

where seed is replaced by an integer number. The network parameters stored in this network realization, including the training parameter $\eta$, can be assessed in this exact function. All figures in the paper were created using the predefined settings and seed 1005 as example out of the results for seeds 1001 to 1020. Table 1 averages over seeds 1001 to 1020. You can use the predefined networks listed here, generate the remaining 10 networks or create new ones, specified by its own seed.

[2] Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, and
Gustavo Batista. The ucr time series classification archive. 2015. URL https://www.cs.ucr.edu/%7Eeamonn/time_series_data_2018/ .

[^1]: Alternative, direct link to dataset: http://www.timeseriesclassification.com/description.php?Dataset=ECG5000

## Theoretical results

The theoretical results presented in fig.2 and fig. 3 of the paper can be created using `figures_2_3.py`. Fig. 2 is created using the function `fig2`, fig. 3 using the functions `optimize_soft_margins`, `random_u_soft_margins` and `fig3`. To generate fig. 2, you can use the the lower case letters "a", "b", "d" to generate the data for fig. 2 a, b, and d, respectively. Fig. 2c is re-using the data created for fig. 2b. Capital letters "A", "B", "C", "D" can be used to plot the corresponding subfigures. "oneline" places all subfigures next to each other, "variate\_u" and "variate\_mu" refer to whether variation over realizations of mu or variation over random input vectors is shown and are explained at the beginning of the function `fig2`. You can also leave both out, but data created using "variate\_u" must be evaluated using the same keyword.

You can use this to start calculating subfigures a to d in parallel like, for example

```fig2ad
python -c 'import figures_2_3 as figs; figs.fig2(seed, "a", "d", "A", "D", "oneline", "variate_u")'
```

and 

```fig2bc
python -c 'import figures_2_3 as figs; figs.fig2(seed, "b", "A", "B", "C", "D", "oneline", "variate_u")'
```

and recieve fig. 2 (a,d) while the second command is still running. As the second command finishes, it will find the results from a and d and print the whole figure. Note: The example data provided use the seed 1005 and the option "variate\_u". To use the option "variate\_mu" or neither of them, data has to be created using the correct option.
The data for fig. 3b are generated with the functions `optimize_soft_margins` and `random_u_soft_margins`. The former finds the optimal solution for the sample stimuli and can be started individually for each $\|\mu\|$, which is passed to the function as length. In the same way, the results for random input vectors are added using `random_u_soft_margins`. For example,

```fig3bopt
python -c 'import figures_2_3 as figs; figs.optimize_soft_margins(seed, 1.0); figs.random_u_soft_margins(seed, 1.0)'
```

for $\|\mu\|=1.0$. All lengths given in the network initialization have to be optimized (see function `setup_network` or import an initialization from `data/network_realizations/` and read out `network_dictionary['lengths']`). Once these are finished, fig. 3 can be created using

```fig3
python -c 'import figures_2_3 as figs; figs.fig3(seed, "a", "A", "B")'
```

using the same notation of lower and upper case letters as in figure 2.

Results are stored in `data/linear_networks/` for fig. 2 and `data/responses_soft_margins/` for fig. 3. For each figure, example results as well as figures are provided.

## Training on dataset ECG5000

The ECG5000 dataset is trained using `ECG_train.py`. It requires a seed, a system type (lin for linear reservoir or non for nonlinear reservoir) and can optionally handle an integer n\_rand (number of random input vectors used for comparison) and the name of a solver for determination of the readout vector (currently, eigenvalue or ridge for ridge regression; The figures in the paper were created using eigenvalue). Results are stored at `data/ECG/`. An example use could be

```train
python ECG_train.py 1001 lin 50 eigenvalue
```

Examples of training results are already stored at `data/ECG` for $\eta=10$ and at `data/ECG/additional/` for $\eta=30$. In the analysis of our paper, these showed to be a reasonable range for this parameter. Higher values result in a stronger suppression of noise in readout direction, smaller values mainly optimize the absolute distance between the class centers. For Gaussian distributed data, $\eta\rightarrow\infty$ corresponds to the limiting case, where the soft margin transforms into the margin; However, the less close to a Gaussian distribution the network states are, the weaker is the approximation of the soft margin (eq. 6 in the paper) for large $\eta$. This is due to the contributions of higher order cumulants. However, as estimator of the rough shape of the resulting network state distribution, an optimization with a large $\eta$ may anyway lead to a good classification performance.

## Evaluation of ECG5000 data

To obtain table 1 and fig. 4 of the paper, use `ECG_eval.py`.
It requires a seed and type for fig. 4, where the type can be lin, non (for linear or non-linear figs. a and b, respectively), nonlin (for a combination of fig. a, b for both linear and non-linear reservoir) or lincomp or noncomp (to generade fig. 4 a-d including composition of the soft margin as in the paper; the presented figure uses noncomp). Afterwards, any number of seeds can be listed to include in tables such as Table 1 of the paper. For example

```eval
python ECG_evaluation 1015 noncomp 1011 1012 1013 1014 1015
```

creates fig. 4, stored at `data/ECG/`, for network seed 1015 and prints table 1 for the five seeds listed at the end. The corresponding training results must be stored in `data/ECG/` beforehand as explained in the previous section. This function gives you some additional tables to inspect the gain from optimization or non-linearity seed-wise.

## Results

Our method significantly increases the separation of network state clouds for the different classes while maintaining a low spatial extension of each cloud (see fig. 4 in the paper).


## Contributing

> 📋This will be filled out at a later stage of the publication process.
>
> Pick a licence and describe how to contribute to your code repository. 
> Users of this software should include the following citations
> and information in any publication of works that used this
> software.
> Contact person:
> anonymous
