2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Current SDK version is 0.16.6
2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Configure stats pid to 57288
2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from /Users/siemsj/.config/wandb/settings
2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from /Users/siemsj/projects/xlstm/experiments/wandb/settings
2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
2024-09-03 12:53:46,769 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
2024-09-03 12:53:46,770 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'experiments/eval.py', 'program_abspath': '/Users/siemsj/projects/xlstm/experiments/eval.py', 'program': '/Users/siemsj/projects/xlstm/experiments/eval.py'}
2024-09-03 12:53:46,770 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Applying login settings: {}
2024-09-03 12:53:46,770 INFO    MainThread:57288 [wandb_init.py:_log_setup():521] Logging user logs to /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125346-wise-water-249/logs/debug.log
2024-09-03 12:53:46,770 INFO    MainThread:57288 [wandb_init.py:_log_setup():522] Logging internal logs to /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125346-wise-water-249/logs/debug-internal.log
2024-09-03 12:53:46,770 INFO    MainThread:57288 [wandb_init.py:init():561] calling init triggers
2024-09-03 12:53:46,771 INFO    MainThread:57288 [wandb_init.py:init():568] wandb.init called with sweep_config: {}
config: {}
2024-09-03 12:53:46,771 INFO    MainThread:57288 [wandb_init.py:init():611] starting backend
2024-09-03 12:53:46,771 INFO    MainThread:57288 [wandb_init.py:init():615] setting up manager
2024-09-03 12:53:46,774 INFO    MainThread:57288 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=spawn,fork,forkserver, using: spawn
2024-09-03 12:53:46,777 INFO    MainThread:57288 [wandb_init.py:init():623] backend started and connected
2024-09-03 12:53:46,783 INFO    MainThread:57288 [wandb_init.py:init():715] updated telemetry
2024-09-03 12:53:46,799 INFO    MainThread:57288 [wandb_init.py:init():748] communicating run to backend with 90.0 second timeout
2024-09-03 12:53:47,009 ERROR   MainThread:57288 [wandb_init.py:init():774] encountered error: You provided an invalid value for the `resume` argument. The value 'must' is not a valid option for resuming a run (wise-water-249) that does not exist. Please check your inputs and try again with a valid run ID. If you are trying to start a new run, please omit the `resume` argument or use `resume='allow'`.
2024-09-03 12:53:47,101 ERROR   MainThread:57288 [wandb_init.py:init():1199] You provided an invalid value for the `resume` argument. The value 'must' is not a valid option for resuming a run (wise-water-249) that does not exist. Please check your inputs and try again with a valid run ID. If you are trying to start a new run, please omit the `resume` argument or use `resume='allow'`.
Traceback (most recent call last):
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1181, in init
    run = wi.init()
          ^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 780, in init
    raise error
wandb.errors.UsageError: You provided an invalid value for the `resume` argument. The value 'must' is not a valid option for resuming a run (wise-water-249) that does not exist. Please check your inputs and try again with a valid run ID. If you are trying to start a new run, please omit the `resume` argument or use `resume='allow'`.
2024-09-03 12:54:14,619 INFO    MainThread:57288 [wandb_settings.py:_apply_login():1943] Applying login settings: {}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Current SDK version is 0.16.6
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Configure stats pid to 57288
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from /Users/siemsj/.config/wandb/settings
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from /Users/siemsj/projects/xlstm/experiments/wandb/settings
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'experiments/eval.py', 'program_abspath': '/Users/siemsj/projects/xlstm/experiments/eval.py', 'program': '/Users/siemsj/projects/xlstm/experiments/eval.py'}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_setup.py:_flush():76] Applying login settings: {}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:_log_setup():521] Logging user logs to /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125414-wise-water-249/logs/debug.log
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:_log_setup():522] Logging internal logs to /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125414-wise-water-249/logs/debug-internal.log
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:init():561] calling init triggers
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:init():568] wandb.init called with sweep_config: {}
config: {}
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:init():611] starting backend
2024-09-03 12:54:14,622 INFO    MainThread:57288 [wandb_init.py:init():615] setting up manager
2024-09-03 12:54:14,623 INFO    MainThread:57288 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=spawn,fork,forkserver, using: spawn
2024-09-03 12:54:14,623 INFO    MainThread:57288 [wandb_init.py:init():623] backend started and connected
2024-09-03 12:54:14,624 INFO    MainThread:57288 [wandb_init.py:init():715] updated telemetry
2024-09-03 12:54:14,635 INFO    MainThread:57288 [wandb_init.py:init():748] communicating run to backend with 90.0 second timeout
2024-09-03 12:54:32,578 WARNING MainThread:57288 [wandb_init.py:init():1203] interrupted
Traceback (most recent call last):
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1181, in init
    run = wi.init()
          ^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 751, in init
    result = run_init_handle.wait(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 283, in wait
    found, abandoned = self._slot._get_and_clear(timeout=wait_timeout)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 130, in _get_and_clear
    if self._wait(timeout=timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/lib/mailbox.py", line 126, in _wait
    return self._event.wait(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/threading.py", line 629, in wait
    signaled = self._cond.wait(timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/threading.py", line 331, in wait
    gotit = waiter.acquire(True, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
