2024-09-03 13:52:51,927 INFO    StreamThr :59318 [internal.py:wandb_internal():86] W&B internal server running at pid: 59318, started at: 2024-09-03 13:52:51.926701
2024-09-03 13:52:51,929 DEBUG   HandlerThread:59318 [handler.py:handle_request():146] handle_request: status
2024-09-03 13:52:51,937 INFO    WriterThread:59318 [datastore.py:open_for_write():87] open: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_135251-wise-water-249/run-wise-water-249.wandb
2024-09-03 13:52:51,938 DEBUG   SenderThread:59318 [sender.py:send():379] send: header
2024-09-03 13:52:51,955 DEBUG   SenderThread:59318 [sender.py:send():379] send: run
2024-09-03 13:52:51,958 INFO    SenderThread:59318 [sender.py:_setup_resume():749] checking resume status for tabpfn_interpretability/xlstm-training/wise-water-249
2024-09-03 13:52:52,580 INFO    SenderThread:59318 [dir_watcher.py:__init__():211] watching files in: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_135251-wise-water-249/files
2024-09-03 13:52:52,580 INFO    SenderThread:59318 [sender.py:_start_run_threads():1124] run started: wise-water-249 with start time 1725364371.927069
2024-09-03 13:52:52,590 DEBUG   HandlerThread:59318 [handler.py:handle_request():146] handle_request: check_version
2024-09-03 13:52:52,590 DEBUG   SenderThread:59318 [sender.py:send_request():406] send_request: check_version
2024-09-03 13:52:52,678 DEBUG   HandlerThread:59318 [handler.py:handle_request():146] handle_request: run_start
2024-09-03 13:52:52,685 DEBUG   HandlerThread:59318 [system_info.py:__init__():26] System info init
2024-09-03 13:52:52,685 DEBUG   HandlerThread:59318 [system_info.py:__init__():41] System info init done
2024-09-03 13:52:52,685 INFO    HandlerThread:59318 [system_monitor.py:start():194] Starting system monitor
2024-09-03 13:52:52,685 INFO    SystemMonitor:59318 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-09-03 13:52:52,686 INFO    SystemMonitor:59318 [interfaces.py:start():190] Started cpu monitoring
2024-09-03 13:52:52,686 INFO    HandlerThread:59318 [system_monitor.py:probe():214] Collecting system info
2024-09-03 13:52:52,687 INFO    SystemMonitor:59318 [interfaces.py:start():190] Started disk monitoring
2024-09-03 13:52:52,688 INFO    SystemMonitor:59318 [interfaces.py:start():190] Started gpuapple monitoring
2024-09-03 13:52:52,688 INFO    SystemMonitor:59318 [interfaces.py:start():190] Started memory monitoring
2024-09-03 13:52:52,689 DEBUG   HandlerThread:59318 [system_info.py:probe():150] Probing system
2024-09-03 13:52:52,689 INFO    SystemMonitor:59318 [interfaces.py:start():190] Started network monitoring
2024-09-03 13:52:52,695 DEBUG   HandlerThread:59318 [system_info.py:_probe_git():135] Probing git
2024-09-03 13:52:52,710 DEBUG   HandlerThread:59318 [system_info.py:_probe_git():143] Probing git done
2024-09-03 13:52:52,710 DEBUG   HandlerThread:59318 [system_info.py:probe():198] Probing system done
2024-09-03 13:52:52,710 DEBUG   HandlerThread:59318 [system_monitor.py:probe():223] {'os': 'macOS-14.6.1-arm64-arm-64bit', 'python': '3.11.8', 'heartbeatAt': '2024-09-03T11:52:52.689548', 'startedAt': '2024-09-03T11:52:51.913998', 'docker': None, 'cuda': None, 'args': ('--config', '/Users/siemsj/projects/xlstm/saved_models_simple_recurrent_20240903_123658_seed_42/config.yaml', '--model', '/Users/siemsj/projects/xlstm/saved_models_simple_recurrent_20240903_123658_seed_42/model_simple_recurrent_seed_42.pth', '--wandb_run_id', 'wise-water-249'), 'state': 'running', 'program': '/Users/siemsj/projects/xlstm/experiments/eval.py', 'codePathLocal': 'eval.py', 'codePath': 'experiments/eval.py', 'git': {'remote': 'git@github.com:JulienSiems/xlstm_mod.git', 'commit': '6c95cfb0c50072c521d10a61150daf6097fbb296'}, 'email': 'juliensiems@gmail.com', 'root': '/Users/siemsj/projects/xlstm', 'host': 'mladms-MacBook-Air.local', 'username': 'siemsj', 'executable': '/Users/siemsj/miniconda3/envs/mother_gam/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'disk': {'/': {'total': 460.4317207336426, 'used': 278.1758346557617}}, 'gpuapple': {'type': 'arm', 'vendor': 'Apple'}, 'memory': {'total': 16.0}}
2024-09-03 13:52:52,710 INFO    HandlerThread:59318 [system_monitor.py:probe():224] Finished collecting system info
2024-09-03 13:52:52,710 INFO    HandlerThread:59318 [system_monitor.py:probe():227] Publishing system info
2024-09-03 13:52:52,710 DEBUG   HandlerThread:59318 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-09-03 13:52:53,008 ERROR   gpuapple  :59318 [gpu_apple.py:sample():81] GPU stats error: Command '['/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/bin/apple_gpu_stats', '--json']' died with <Signals.SIGINT: 2>.
Traceback (most recent call last):
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/internal/system/assets/gpu_apple.py", line 60, in sample
    subprocess.check_output(command, universal_newlines=True)
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/bin/apple_gpu_stats', '--json']' died with <Signals.SIGINT: 2>.
