2024-09-03 12:55:11,133 INFO    StreamThr :57374 [internal.py:wandb_internal():86] W&B internal server running at pid: 57374, started at: 2024-09-03 12:55:11.132953
2024-09-03 12:55:11,135 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status
2024-09-03 12:55:11,139 INFO    WriterThread:57374 [datastore.py:open_for_write():87] open: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/run-wise-water-249.wandb
2024-09-03 12:55:11,140 DEBUG   SenderThread:57374 [sender.py:send():379] send: header
2024-09-03 12:55:11,157 DEBUG   SenderThread:57374 [sender.py:send():379] send: run
2024-09-03 12:55:11,170 INFO    SenderThread:57374 [sender.py:_setup_resume():749] checking resume status for tabpfn_interpretability/xlstm-experiments/wise-water-249
2024-09-03 12:55:11,401 INFO    SenderThread:57374 [sender.py:_setup_resume():827] configured resuming with: ResumeState(resumed=True,step=0,history=0,events=0,output=0,runtime=0,wandb_runtime=None,summary={},config={'_wandb': {'desc': None, 'value': {'t': {'1': [1, 11, 49, 55], '3': [14, 23], '4': '3.11.8', '5': '0.16.6', '6': '4.42.4', '8': [5, 13], '13': 'darwin-arm64'}, 'framework': 'huggingface', 'start_time': 1725360854.0, 'cli_version': '0.16.6', 'is_jupyter_run': False, 'python_version': '3.11.8', 'is_kaggle_kernel': False, 'huggingface_version': '4.42.4'}}},tags=[])
2024-09-03 12:55:11,807 INFO    SenderThread:57374 [dir_watcher.py:__init__():211] watching files in: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/files
2024-09-03 12:55:11,807 INFO    SenderThread:57374 [sender.py:_start_run_threads():1124] run started: wise-water-249 with start time 1725360911.133443
2024-09-03 12:55:11,834 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: check_version
2024-09-03 12:55:11,834 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: check_version
2024-09-03 12:55:11,932 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: run_start
2024-09-03 12:55:11,940 DEBUG   HandlerThread:57374 [system_info.py:__init__():26] System info init
2024-09-03 12:55:11,940 DEBUG   HandlerThread:57374 [system_info.py:__init__():41] System info init done
2024-09-03 12:55:11,940 INFO    HandlerThread:57374 [system_monitor.py:start():194] Starting system monitor
2024-09-03 12:55:11,940 INFO    SystemMonitor:57374 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-09-03 12:55:11,941 INFO    SystemMonitor:57374 [interfaces.py:start():190] Started cpu monitoring
2024-09-03 12:55:11,943 INFO    SystemMonitor:57374 [interfaces.py:start():190] Started disk monitoring
2024-09-03 12:55:11,943 INFO    SystemMonitor:57374 [interfaces.py:start():190] Started gpuapple monitoring
2024-09-03 12:55:11,944 INFO    SystemMonitor:57374 [interfaces.py:start():190] Started memory monitoring
2024-09-03 12:55:11,945 INFO    SystemMonitor:57374 [interfaces.py:start():190] Started network monitoring
2024-09-03 12:55:12,476 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: python_packages
2024-09-03 12:55:12,476 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: python_packages
2024-09-03 12:55:12,477 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 12:55:12,477 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: stop_status
2024-09-03 12:55:12,480 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 12:55:12,725 DEBUG   SenderThread:57374 [sender.py:send():379] send: telemetry
2024-09-03 12:55:12,817 INFO    Thread-16 :57374 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/files/requirements.txt
2024-09-03 12:55:16,742 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:21,754 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:23,853 INFO    Thread-16 :57374 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/files/output.log
2024-09-03 12:55:25,862 INFO    Thread-16 :57374 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/files/output.log
2024-09-03 12:55:27,080 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:27,478 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 12:55:27,479 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: stop_status
2024-09-03 12:55:27,486 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 12:55:32,680 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:37,700 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:42,480 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 12:55:42,480 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: stop_status
2024-09-03 12:55:42,483 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 12:55:42,708 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:42,919 INFO    Thread-16 :57374 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_125511-wise-water-249/files/config.yaml
2024-09-03 12:55:47,901 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:52,919 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 12:55:57,485 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 12:55:57,486 DEBUG   HandlerThread:57374 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 12:55:57,486 DEBUG   SenderThread:57374 [sender.py:send_request():406] send_request: stop_status
2024-09-03 12:55:58,006 ERROR   gpuapple  :57374 [gpu_apple.py:sample():81] GPU stats error: Command '['/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/bin/apple_gpu_stats', '--json']' died with <Signals.SIGINT: 2>.
Traceback (most recent call last):
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/sdk/internal/system/assets/gpu_apple.py", line 60, in sample
    subprocess.check_output(command, universal_newlines=True)
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/siemsj/miniconda3/envs/mother_gam/lib/python3.11/site-packages/wandb/bin/apple_gpu_stats', '--json']' died with <Signals.SIGINT: 2>.
