2024-09-03 13:46:47,360 INFO    StreamThr :59096 [internal.py:wandb_internal():86] W&B internal server running at pid: 59096, started at: 2024-09-03 13:46:47.359847
2024-09-03 13:46:47,362 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status
2024-09-03 13:46:47,364 INFO    WriterThread:59096 [datastore.py:open_for_write():87] open: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/run-wise-water-249.wandb
2024-09-03 13:46:47,367 DEBUG   SenderThread:59096 [sender.py:send():379] send: header
2024-09-03 13:46:47,383 DEBUG   SenderThread:59096 [sender.py:send():379] send: run
2024-09-03 13:46:47,397 INFO    SenderThread:59096 [sender.py:_setup_resume():749] checking resume status for tabpfn_interpretability/xlstm-experiments/wise-water-249
2024-09-03 13:46:47,649 INFO    SenderThread:59096 [sender.py:_setup_resume():827] configured resuming with: ResumeState(resumed=True,step=0,history=0,events=16,output=123,runtime=483.982519,wandb_runtime=None,summary={},config={'_wandb': {'desc': None, 'value': {'t': {'1': [1, 11, 49, 55], '2': [1, 11, 49, 55], '3': [5, 14, 23, 62], '4': '3.11.8', '5': '0.16.6', '6': '4.42.4', '8': [4, 5], '13': 'darwin-arm64'}, 'framework': 'huggingface', 'start_time': 1725363563.0, 'cli_version': '0.16.6', 'is_jupyter_run': False, 'python_version': '3.11.8', 'is_kaggle_kernel': False, 'huggingface_version': '4.42.4'}}},tags=[])
2024-09-03 13:46:47,991 INFO    SenderThread:59096 [dir_watcher.py:__init__():211] watching files in: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files
2024-09-03 13:46:47,991 INFO    SenderThread:59096 [sender.py:_start_run_threads():1124] run started: wise-water-249 with start time 1725363523.377582
2024-09-03 13:46:48,009 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: check_version
2024-09-03 13:46:48,010 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: check_version
2024-09-03 13:46:48,104 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: run_start
2024-09-03 13:46:48,112 DEBUG   HandlerThread:59096 [system_info.py:__init__():26] System info init
2024-09-03 13:46:48,112 DEBUG   HandlerThread:59096 [system_info.py:__init__():41] System info init done
2024-09-03 13:46:48,112 INFO    HandlerThread:59096 [system_monitor.py:start():194] Starting system monitor
2024-09-03 13:46:48,113 INFO    SystemMonitor:59096 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-09-03 13:46:48,114 INFO    SystemMonitor:59096 [interfaces.py:start():190] Started cpu monitoring
2024-09-03 13:46:48,115 INFO    SystemMonitor:59096 [interfaces.py:start():190] Started disk monitoring
2024-09-03 13:46:48,116 INFO    SystemMonitor:59096 [interfaces.py:start():190] Started gpuapple monitoring
2024-09-03 13:46:48,117 INFO    SystemMonitor:59096 [interfaces.py:start():190] Started memory monitoring
2024-09-03 13:46:48,118 INFO    SystemMonitor:59096 [interfaces.py:start():190] Started network monitoring
2024-09-03 13:46:48,483 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: python_packages
2024-09-03 13:46:48,483 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: python_packages
2024-09-03 13:46:48,484 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:46:48,484 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:46:48,485 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:46:48,670 DEBUG   SenderThread:59096 [sender.py:send():379] send: telemetry
2024-09-03 13:46:48,999 INFO    Thread-16 :59096 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files/output.log
2024-09-03 13:46:48,999 INFO    Thread-16 :59096 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files/requirements.txt
2024-09-03 13:46:50,996 INFO    Thread-16 :59096 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files/output.log
2024-09-03 13:46:52,684 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:46:57,699 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:02,717 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:03,489 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:47:03,489 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:47:03,490 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:47:07,996 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:13,009 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:18,028 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:18,490 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:47:18,492 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:47:18,492 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:47:19,096 INFO    Thread-16 :59096 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files/config.yaml
2024-09-03 13:47:23,738 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:28,749 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:33,495 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:47:33,497 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:47:33,498 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:47:33,839 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:38,856 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:43,874 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:48,125 DEBUG   SystemMonitor:59096 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-09-03 13:47:48,127 DEBUG   SenderThread:59096 [sender.py:send():379] send: telemetry
2024-09-03 13:47:48,129 DEBUG   SenderThread:59096 [sender.py:send():379] send: stats
2024-09-03 13:47:48,500 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:47:48,502 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:47:48,502 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:47:49,710 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:50,208 INFO    Thread-16 :59096 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_134647-wise-water-249/files/config.yaml
2024-09-03 13:47:54,931 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:47:59,953 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 13:48:03,505 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 13:48:03,508 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 13:48:03,508 DEBUG   SenderThread:59096 [sender.py:send_request():406] send_request: stop_status
2024-09-03 13:48:05,679 DEBUG   HandlerThread:59096 [handler.py:handle_request():146] handle_request: status_report
