# NeurIPS 2023 Submission 2667:  Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training



## Environment and Dataset Setup

```bash
# install environment
conda create -n ww_train python=3.8
conda activate ww_train
pip install -r requirements.txt

# install TinyImageNet
bash dataset_setup.sh
```



## Main Result 1:  Compare TempBalance (TB) to CAL and  SNR (Figure 3) 

### CIFAR100 / CIFAR10/ SVHN
```bash
cd neurips_2023_tempbalance
# CAL baseline
bash src/scripts/train_baseline.sh

# SNR baseline
bash src/scripts/train_snr.sh

# Our method TB / TB + SNR  
bash src/scripts/train_tb.sh
```
### TIN

```bash
#CAL baseline
bash src/scripts/tinyimagenet_train_baseline.sh

#SNR baseline
bash src/scripts/tinyimagenet_train_snr.sh

#Our method TB / TB + SNR   
bash src/scripts/tinyimagenet_train_tb.sh
```



## Main Result 2: compare TB and baseline optimizers.  (Figure 4)

```bash
# CAL baseline (same as Main Result 1)
bash src/scripts/train_baseline.sh

# SGDR
bash src/scripts/cifar100_train_baseline_more_lr_schedule.sh

# LARS
bash src/scripts/cifar100_train_baseline_moreopt_lars.sh

# Lookahead
bash src/scripts/cifar100_train_baseline_moreopt.sh

# SGDP
bash src/scripts/cifar100_train_baseline_moreopt_sgdp.sh

# TB          (same as Main Result 1)
bash src/scripts/train_tb.sh

# TB + SGDP
bash src/scripts/cifar100_train_tb_sgdp.sh
```



## Ablation Study 1: tuning learning rate. (Figure 5)

```bash
# CAL baseline
bash src/scripts/train_baseline_lr_abl.sh
# TB 
bash src/scripts/train_tb_lr_abl.sh
```

## Ablation Study 2: different model width (Figure 6)

```bash
# CAL baseline
bash src/scripts/train_baseline_width_abl.sh
# TB
bash src/scripts/train_tb_width_abl.sh
```

## Ablation Study 3: different HT-SR metrics (Figure 7)

```bash
# Spectral Norm / AlphaWeighted
bash src/scripts/train_tb_metric_abl.sh

# PL_Alpha_Hill (same as Main Result 1)
bash src/scripts/train_tb.sh
```

