2024-09-03 17:34:39,260 INFO    StreamThr :64411 [internal.py:wandb_internal():86] W&B internal server running at pid: 64411, started at: 2024-09-03 17:34:39.259664
2024-09-03 17:34:39,261 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status
2024-09-03 17:34:39,267 INFO    WriterThread:64411 [datastore.py:open_for_write():87] open: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/run-wo08hjyp.wandb
2024-09-03 17:34:39,268 DEBUG   SenderThread:64411 [sender.py:send():379] send: header
2024-09-03 17:34:39,285 DEBUG   SenderThread:64411 [sender.py:send():379] send: run
2024-09-03 17:34:39,286 INFO    SenderThread:64411 [sender.py:_setup_resume():749] checking resume status for tabpfn_interpretability/xlstm-training/wo08hjyp
2024-09-03 17:34:39,554 INFO    SenderThread:64411 [sender.py:_setup_resume():827] configured resuming with: ResumeState(resumed=True,step=31,history=31,events=50,output=199,runtime=1411.945041,wandb_runtime=43,summary={'_runtime': 1411.940847158432, '_step': 30, '_timestamp': 1725377545.0957792, '_wandb': {'runtime': 43}, 'accuracy_vs_sequence_length': {'_type': 'image-file', 'format': 'png', 'height': 601, 'path': 'media/images/accuracy_vs_sequence_length_30_4aa3fb72a345039d8ad8.png', 'sha256': '4aa3fb72a345039d8ad8a24069f61ced3a75823146b8b8381495753ad03c0798', 'size': 107706, 'width': 1001}, 'avg_accuracy_vs_sequence_length': {'_type': 'image-file', 'format': 'png', 'height': 600, 'path': 'media/images/avg_accuracy_vs_sequence_length_22_7d5cdf1f5a0d9f85f265.png', 'sha256': '7d5cdf1f5a0d9f85f2650e14a50eabe8ee688e84e96e3ccf8a9cce0edfe3cde4', 'size': 164535, 'width': 1000}, 'epoch': 1, 'model_size_vs_sequence_length': {'_type': 'image-file', 'format': 'png', 'height': 801, 'path': 'media/images/model_size_vs_sequence_length_29_12fdcb5db0194824fa0b.png', 'sha256': '12fdcb5db0194824fa0b792781b259bbe275f689244c50843f96595c87e07700', 'size': 201233, 'width': 1199}, 'performance_analysis': {'_type': 'image-file', 'format': 'png', 'height': 779, 'path': 'media/images/performance_analysis_28_e15efa5a6e3a0716b494.png', 'sha256': 'e15efa5a6e3a0716b4942c775f1f6297417a549d490fdd2e0ce638b90b44951f', 'size': 209269, 'width': 1980}, 'sequence_length_vs_accuracy': {'_type': 'image-file', 'format': 'png', 'height': 600, 'path': 'media/images/sequence_length_vs_accuracy_21_1e6a33960a5e7b327af1.png', 'sha256': '1e6a33960a5e7b327af12317dee8df4f4137154279b8279676f35caa95b08351', 'size': 26577, 'width': 1000}, 'step': 2000, 'test_accuracy': 0.8784423828125, 'test_loss': 0.38031005859375, 'train_SequenceAccuracy': 0.5937871336936951, 'train_loss': 0.901184554763136, 'val_validation_SequenceAccuracy': 0.877685546875, 'val_validation_loss': 0.37054199911654},config={'model': {'desc': None, 'value': {'name': 'simple_recurrent', 'dropout': 0, 'layer_type': 'diagonal', 'num_blocks': 1, 'vocab_size': 3, 'tie_weights': False, 'embedding_dim': 3, 'context_length': 256, 'activation_func': 'tanh', 'add_embedding_dropout': False, 'weight_decay_on_embedding': False}}, '_wandb': {'desc': None, 'value': {'t': {'1': [1, 11, 49, 55], '2': [1, 11, 49, 55], '3': [5, 14, 23, 62], '4': '3.11.8', '5': '0.16.6', '6': '4.42.4', '8': [4, 5], '13': 'darwin-arm64'}, 'framework': 'huggingface', 'start_time': 1725377515.0, 'cli_version': '0.16.6', 'is_jupyter_run': False, 'python_version': '3.11.8', 'is_kaggle_kernel': False, 'huggingface_version': '4.42.4'}}, 'dataset': {'desc': None, 'value': {'name': 'form_language', 'kwargs': {'seed': 42, 'count': {'test': 8192, 'train': 5120000, 'validation': 8192}, 'subpar': {'test': {'max_sequence_length': 256, 'min_sequence_length': 40}, 'validation': {'max_sequence_length': 256, 'min_sequence_length': 40}}, 'vocab_size': 3, 'enable_mask': True, 'context_length': 256, 'synth_lang_type': 'parity', 'max_sequence_length': 40, 'min_sequence_length': 3}}}, 'training': {'desc': None, 'value': {'lr': 0.001, 'seed': 42, 'device': 'cpu', 'compile': True, 'num_steps': 2000, 'batch_size': 256, 'weight_decay': 0, 'amp_precision': 'bfloat16', 'grad_clip_norm': 'None', 'val_every_step': 200, 'lr_decay_factor': 0.001, 'lr_warmup_steps': 2000, 'weight_precision': 'float32', 'lr_decay_until_steps': 2000, 'enable_mixed_precision': True}}, 'dtype_used': {'desc': None, 'value': 'torch.bfloat16'}},tags=[])
2024-09-03 17:34:39,870 INFO    SenderThread:64411 [dir_watcher.py:__init__():211] watching files in: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files
2024-09-03 17:34:39,870 INFO    SenderThread:64411 [sender.py:_start_run_threads():1124] run started: wo08hjyp with start time 1725376267.315492
2024-09-03 17:34:39,873 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: summary_record
2024-09-03 17:34:39,873 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:34:39,876 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: check_version
2024-09-03 17:34:39,877 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: check_version
2024-09-03 17:34:39,970 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: run_start
2024-09-03 17:34:39,977 DEBUG   HandlerThread:64411 [system_info.py:__init__():26] System info init
2024-09-03 17:34:39,977 DEBUG   HandlerThread:64411 [system_info.py:__init__():41] System info init done
2024-09-03 17:34:39,977 INFO    HandlerThread:64411 [system_monitor.py:start():194] Starting system monitor
2024-09-03 17:34:39,977 INFO    SystemMonitor:64411 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-09-03 17:34:39,978 INFO    SystemMonitor:64411 [interfaces.py:start():190] Started cpu monitoring
2024-09-03 17:34:39,979 INFO    SystemMonitor:64411 [interfaces.py:start():190] Started disk monitoring
2024-09-03 17:34:39,981 INFO    SystemMonitor:64411 [interfaces.py:start():190] Started gpuapple monitoring
2024-09-03 17:34:39,982 INFO    SystemMonitor:64411 [interfaces.py:start():190] Started memory monitoring
2024-09-03 17:34:39,983 INFO    SystemMonitor:64411 [interfaces.py:start():190] Started network monitoring
2024-09-03 17:34:40,131 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: python_packages
2024-09-03 17:34:40,131 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 17:34:40,131 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 17:34:40,131 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: python_packages
2024-09-03 17:34:40,132 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: stop_status
2024-09-03 17:34:40,290 DEBUG   SenderThread:64411 [sender.py:send():379] send: telemetry
2024-09-03 17:34:40,873 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:34:40,873 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:34:40,873 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/requirements.txt
2024-09-03 17:34:42,882 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:34:44,306 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:34:49,319 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:34:54,335 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:34:55,132 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 17:34:55,132 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: stop_status
2024-09-03 17:34:55,136 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 17:34:58,953 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:34:59,683 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:35:01,292 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: partial_history
2024-09-03 17:35:01,292 DEBUG   SenderThread:64411 [sender.py:send():379] send: files
2024-09-03 17:35:01,292 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png with policy now
2024-09-03 17:35:01,292 DEBUG   SenderThread:64411 [sender.py:send():379] send: history
2024-09-03 17:35:01,293 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: summary_record
2024-09-03 17:35:01,293 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:35:01,965 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:35:01,965 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png
2024-09-03 17:35:01,966 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media
2024-09-03 17:35:01,966 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images
2024-09-03 17:35:02,370 INFO    wandb-upload_0:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/rtr5thwz-media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png
2024-09-03 17:35:02,971 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:35:03,477 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: partial_history
2024-09-03 17:35:03,477 DEBUG   SenderThread:64411 [sender.py:send():379] send: files
2024-09-03 17:35:03,477 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png with policy now
2024-09-03 17:35:03,478 DEBUG   SenderThread:64411 [sender.py:send():379] send: history
2024-09-03 17:35:03,478 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: summary_record
2024-09-03 17:35:03,478 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:35:03,976 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:35:03,976 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png
2024-09-03 17:35:03,976 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images
2024-09-03 17:35:04,976 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:35:05,484 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:35:05,541 INFO    wandb-upload_1:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/cdkddu85-media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png
2024-09-03 17:35:06,001 INFO    wandb-upload_0:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/t8dol62m-media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png
2024-09-03 17:35:06,465 INFO    wandb-upload_1:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/se51l2sh-media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png
2024-09-03 17:35:08,752 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: partial_history
2024-09-03 17:35:08,753 DEBUG   SenderThread:64411 [sender.py:send():379] send: files
2024-09-03 17:35:08,753 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png with policy now
2024-09-03 17:35:08,754 DEBUG   SenderThread:64411 [sender.py:send():379] send: history
2024-09-03 17:35:08,754 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: summary_record
2024-09-03 17:35:08,754 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:35:08,754 DEBUG   SenderThread:64411 [sender.py:send():379] send: exit
2024-09-03 17:35:08,754 INFO    SenderThread:64411 [sender.py:send_exit():586] handling exit code: 0
2024-09-03 17:35:08,754 INFO    SenderThread:64411 [sender.py:send_exit():588] handling runtime: 71
2024-09-03 17:35:08,754 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:35:08,754 INFO    SenderThread:64411 [sender.py:send_exit():594] send defer
2024-09-03 17:35:08,755 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:08,755 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 0
2024-09-03 17:35:08,755 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:08,755 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 0
2024-09-03 17:35:08,755 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 1
2024-09-03 17:35:08,755 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:08,755 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 1
2024-09-03 17:35:08,755 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:08,755 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 1
2024-09-03 17:35:08,755 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 2
2024-09-03 17:35:08,755 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:08,755 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 2
2024-09-03 17:35:08,755 INFO    HandlerThread:64411 [system_monitor.py:finish():203] Stopping system monitor
2024-09-03 17:35:08,755 DEBUG   SystemMonitor:64411 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-09-03 17:35:08,755 DEBUG   SystemMonitor:64411 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-09-03 17:35:08,755 DEBUG   SystemMonitor:64411 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-09-03 17:35:08,755 INFO    HandlerThread:64411 [interfaces.py:finish():202] Joined cpu monitor
2024-09-03 17:35:08,756 INFO    HandlerThread:64411 [interfaces.py:finish():202] Joined disk monitor
2024-09-03 17:35:08,997 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:35:08,997 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png
2024-09-03 17:35:08,997 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images
2024-09-03 17:35:09,067 INFO    HandlerThread:64411 [interfaces.py:finish():202] Joined gpuapple monitor
2024-09-03 17:35:09,067 INFO    HandlerThread:64411 [interfaces.py:finish():202] Joined memory monitor
2024-09-03 17:35:09,067 INFO    HandlerThread:64411 [interfaces.py:finish():202] Joined network monitor
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 2
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 3
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send():379] send: telemetry
2024-09-03 17:35:09,068 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:09,068 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 3
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 3
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 4
2024-09-03 17:35:09,068 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:09,068 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 4
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 4
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 5
2024-09-03 17:35:09,068 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:09,068 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 5
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send():379] send: summary
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 5
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 6
2024-09-03 17:35:09,068 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:09,068 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 6
2024-09-03 17:35:09,068 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,068 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 6
2024-09-03 17:35:09,069 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 17:35:09,266 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 7
2024-09-03 17:35:09,266 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:09,266 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 7
2024-09-03 17:35:09,266 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:09,266 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 7
2024-09-03 17:35:09,443 INFO    wandb-upload_0:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/oej64ny5-media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png
2024-09-03 17:35:09,759 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: poll_exit
2024-09-03 17:35:10,003 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:35:10,004 INFO    Thread-12 :64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/config.yaml
2024-09-03 17:35:10,257 INFO    wandb-upload_1:64411 [upload_job.py:push():131] Uploaded file /var/folders/gy/2rc7yzfs65v1flwvpgsnln2h0000gp/T/tmplx132b2jwandb/1nukda04-media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png
2024-09-03 17:35:10,525 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 8
2024-09-03 17:35:10,525 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: poll_exit
2024-09-03 17:35:10,525 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:10,526 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 8
2024-09-03 17:35:10,526 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:10,526 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 8
2024-09-03 17:35:10,526 INFO    SenderThread:64411 [job_builder.py:build():318] Attempting to build job artifact
2024-09-03 17:35:10,527 WARNING SenderThread:64411 [job_builder.py:_log_if_verbose():210] Ensure read and write access to run files dir: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files, control this via the WANDB_DIR env var. See https://docs.wandb.ai/guides/track/environment-variables
2024-09-03 17:35:10,527 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 9
2024-09-03 17:35:10,527 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:10,527 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 9
2024-09-03 17:35:10,527 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:10,527 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 9
2024-09-03 17:35:10,527 INFO    SenderThread:64411 [dir_watcher.py:finish():358] shutting down directory watcher
2024-09-03 17:35:10,764 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: poll_exit
2024-09-03 17:35:11,006 INFO    SenderThread:64411 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:35:11,007 INFO    SenderThread:64411 [dir_watcher.py:finish():388] scan: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files
2024-09-03 17:35:11,007 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/requirements.txt requirements.txt
2024-09-03 17:35:11,007 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log output.log
2024-09-03 17:35:11,010 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/config.yaml config.yaml
2024-09-03 17:35:11,012 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json wandb-summary.json
2024-09-03 17:35:11,017 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png media/images/model_size_vs_sequence_length_32_559b9dabb9ed2b3c11c0.png
2024-09-03 17:35:11,017 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png media/images/performance_analysis_31_f0521c2b7ecbf08e6ba1.png
2024-09-03 17:35:11,017 INFO    SenderThread:64411 [dir_watcher.py:finish():402] scan save: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png media/images/accuracy_vs_sequence_length_33_7c8b911f702668a054a2.png
2024-09-03 17:35:11,017 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 10
2024-09-03 17:35:11,017 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: poll_exit
2024-09-03 17:35:11,018 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:11,018 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 10
2024-09-03 17:35:11,019 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:11,019 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 10
2024-09-03 17:35:11,019 INFO    SenderThread:64411 [file_pusher.py:finish():172] shutting down file pusher
2024-09-03 17:35:11,481 INFO    wandb-upload_1:64411 [upload_job.py:push():131] Uploaded file /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/output.log
2024-09-03 17:35:11,491 INFO    wandb-upload_0:64411 [upload_job.py:push():131] Uploaded file /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/requirements.txt
2024-09-03 17:35:11,571 INFO    wandb-upload_2:64411 [upload_job.py:push():131] Uploaded file /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/config.yaml
2024-09-03 17:35:11,598 INFO    wandb-upload_3:64411 [upload_job.py:push():131] Uploaded file /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/files/wandb-summary.json
2024-09-03 17:35:11,769 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: poll_exit
2024-09-03 17:35:11,770 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: poll_exit
2024-09-03 17:35:11,803 INFO    Thread-11 (_thread_body):64411 [sender.py:transition_state():614] send defer: 11
2024-09-03 17:35:11,804 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:11,804 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 11
2024-09-03 17:35:11,805 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:11,805 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 11
2024-09-03 17:35:11,805 INFO    SenderThread:64411 [file_pusher.py:join():178] waiting for file pusher
2024-09-03 17:35:11,808 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 12
2024-09-03 17:35:11,808 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:11,808 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 12
2024-09-03 17:35:11,808 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:11,808 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 12
2024-09-03 17:35:11,809 INFO    SenderThread:64411 [file_stream.py:finish():614] file stream finish called
2024-09-03 17:35:12,091 INFO    SenderThread:64411 [file_stream.py:finish():618] file stream finish is done
2024-09-03 17:35:12,092 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 13
2024-09-03 17:35:12,092 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:12,092 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 13
2024-09-03 17:35:12,093 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:12,093 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 13
2024-09-03 17:35:12,093 INFO    SenderThread:64411 [sender.py:transition_state():614] send defer: 14
2024-09-03 17:35:12,093 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: defer
2024-09-03 17:35:12,093 DEBUG   SenderThread:64411 [sender.py:send():379] send: final
2024-09-03 17:35:12,093 INFO    HandlerThread:64411 [handler.py:handle_request_defer():172] handle defer: 14
2024-09-03 17:35:12,094 DEBUG   SenderThread:64411 [sender.py:send():379] send: footer
2024-09-03 17:35:12,094 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: defer
2024-09-03 17:35:12,094 INFO    SenderThread:64411 [sender.py:send_request_defer():610] handle sender defer: 14
2024-09-03 17:35:12,095 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: poll_exit
2024-09-03 17:35:12,095 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: poll_exit
2024-09-03 17:35:12,095 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: poll_exit
2024-09-03 17:35:12,095 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: server_info
2024-09-03 17:35:12,096 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: poll_exit
2024-09-03 17:35:12,096 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: get_summary
2024-09-03 17:35:12,096 DEBUG   SenderThread:64411 [sender.py:send_request():406] send_request: server_info
2024-09-03 17:35:12,097 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: sampled_history
2024-09-03 17:35:12,099 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 17:35:12,256 INFO    MainThread:64411 [wandb_run.py:_footer_history_summary_info():3936] rendering history
2024-09-03 17:35:12,256 INFO    MainThread:64411 [wandb_run.py:_footer_history_summary_info():3968] rendering summary
2024-09-03 17:35:12,257 INFO    MainThread:64411 [wandb_run.py:_footer_sync_info():3895] logging synced files
2024-09-03 17:35:12,258 DEBUG   HandlerThread:64411 [handler.py:handle_request():146] handle_request: shutdown
2024-09-03 17:35:12,258 INFO    HandlerThread:64411 [handler.py:finish():866] shutting down handler
2024-09-03 17:35:13,100 INFO    WriterThread:64411 [datastore.py:close():296] close: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_173439-wo08hjyp/run-wo08hjyp.wandb
2024-09-03 17:35:13,259 INFO    SenderThread:64411 [sender.py:finish():1546] shutting down sender
2024-09-03 17:35:13,260 INFO    SenderThread:64411 [file_pusher.py:finish():172] shutting down file pusher
2024-09-03 17:35:13,260 INFO    SenderThread:64411 [file_pusher.py:join():178] waiting for file pusher
