# Stochastic Order Evaluation package

Code for NeurIPS 2024 submission "Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking"


## Installation

To install the package, you can use conda to create a new environment with the required dependencies:

```bash
conda env create -f environment.yml
```

Then, activate the environment and install the package:

```bash
conda activate soe
pip install .
```


## Example: univariate relative and absolute stochastic order test

```python
import numpy as np
from soe.testing import StochasticOrderTesting

k = 15  # number of models
n_samples = 100  # number of scores per model

# Generate random scores with a given random mean
means = np.random.permutation(k)
gt_rank = np.argsort(means)[::-1]

# For every model we have its scores in a list
scores_list = [m + np.random.randn(n_samples) for m in means]

test = StochasticOrderTesting(scores_list, n_bootstrap=100)

# Relative stochastic order test
rank_rel_qs, rank_rel_iqs = test.compute_relative_test(alpha=0.05)

# Check rank obtained with quantiles
rank_rel_qs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 

# Check rank obtained with integrated quantiles
rank_rel_iqs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 

# Absolute stochastic order test
rank_abs_qs, rank_abs_iqs = test.compute_absolute_test(alpha=0.05, tau=0.25)

# Check rank obtained with quantiles
rank_abs_qs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 

# Check rank obtained with integrated quantiles
rank_abs_iqs == gt_rank

>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 
```


## Example: multi-variate relative and absolute stochastic order test

```python
import numpy as np
from soe.mvtesting import MVStochasticOrderTesting

k = 3  # number of models
n_resp = 5  # number of simultanesly recorded responses
n_samples = 100  # number of samples (e.g. prompts)

# Generate random scores with a given random mean
means = np.random.permutation(k)
gt_rank = np.argsort(means)[::-1]

# For every model we have its scores in a list
scores_list = [m + np.random.randn(n_samples, n_resp) for m in means]

test = MVStochasticOrderTesting(scores_list, n_bootstrap=100, use_sinkhorn=True, cost='logistic', verbose=True)

# Relative stochastic order test
rank_rel = test.compute_relative_test(alpha=0.05)

# Check rank obtained with relative test quantiles
rank_rel == gt_rank
>>> array([ True,  True,  True])
```


## Juptyer notebooks

You can find more examples in the `scripts` folder.
* `scripts/synthetic_test.ipynb` demonostrate the use of first-order and second-order stochastic dominance tests in the univariate case on a synthetically generated dataset.
* `scripts/synthetic_test_mv.ipynb` demonostrate the use of multivariate first-order stochastic dominance tests on the synthetically generated dataset analyzed in the NeurIPS 2024 submission, and generates Figure 1.
