2024-09-03 14:17:34,061 INFO    StreamThr :59848 [internal.py:wandb_internal():86] W&B internal server running at pid: 59848, started at: 2024-09-03 14:17:34.060322
2024-09-03 14:17:34,063 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status
2024-09-03 14:17:34,071 INFO    WriterThread:59848 [datastore.py:open_for_write():87] open: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/run-wo08hjyp.wandb
2024-09-03 14:17:34,072 DEBUG   SenderThread:59848 [sender.py:send():379] send: header
2024-09-03 14:17:34,088 DEBUG   SenderThread:59848 [sender.py:send():379] send: run
2024-09-03 14:17:34,091 INFO    SenderThread:59848 [sender.py:_setup_resume():749] checking resume status for tabpfn_interpretability/xlstm-training/wo08hjyp
2024-09-03 14:17:34,398 INFO    SenderThread:59848 [sender.py:_setup_resume():827] configured resuming with: ResumeState(resumed=True,step=21,history=21,events=13,output=31,runtime=369.016128,wandb_runtime=366,summary={'_runtime': 369.013386964798, '_step': 20, '_timestamp': 1725365620.67517, '_wandb': {'runtime': 366}, 'epoch': 1, 'step': 2000, 'test_accuracy': 0.860595703125, 'test_loss': 0.41156005859375, 'train_SequenceAccuracy': 0.5937871336936951, 'train_loss': 0.901184554763136, 'val_validation_SequenceAccuracy': 0.877685546875, 'val_validation_loss': 0.37054199911654},config={'model': {'desc': None, 'value': {'name': 'simple_recurrent', 'dropout': 0, 'layer_type': 'diagonal', 'num_blocks': 1, 'vocab_size': 3, 'tie_weights': False, 'embedding_dim': 3, 'context_length': 256, 'activation_func': 'tanh', 'add_embedding_dropout': False, 'weight_decay_on_embedding': False}}, '_wandb': {'desc': None, 'value': {'t': {'1': [1, 11, 49, 55], '2': [1, 11, 49, 55], '3': [5, 14, 23, 62], '4': '3.11.8', '5': '0.16.6', '6': '4.42.4', '8': [4, 5], '13': 'darwin-arm64'}, 'framework': 'huggingface', 'start_time': 1725365618.0, 'cli_version': '0.16.6', 'is_jupyter_run': False, 'python_version': '3.11.8', 'is_kaggle_kernel': False, 'huggingface_version': '4.42.4'}}, 'dataset': {'desc': None, 'value': {'name': 'form_language', 'kwargs': {'seed': 42, 'count': {'test': 8192, 'train': 5120000, 'validation': 8192}, 'subpar': {'test': {'max_sequence_length': 256, 'min_sequence_length': 40}, 'validation': {'max_sequence_length': 256, 'min_sequence_length': 40}}, 'vocab_size': 3, 'enable_mask': True, 'context_length': 256, 'synth_lang_type': 'parity', 'max_sequence_length': 40, 'min_sequence_length': 3}}}, 'training': {'desc': None, 'value': {'lr': 0.001, 'seed': 42, 'device': 'cpu', 'compile': True, 'num_steps': 2000, 'batch_size': 256, 'weight_decay': 0, 'amp_precision': 'bfloat16', 'grad_clip_norm': 'None', 'val_every_step': 200, 'lr_decay_factor': 0.001, 'lr_warmup_steps': 2000, 'weight_precision': 'float32', 'lr_decay_until_steps': 2000, 'enable_mixed_precision': True}}, 'dtype_used': {'desc': None, 'value': 'torch.bfloat16'}},tags=[])
2024-09-03 14:17:34,746 INFO    SenderThread:59848 [dir_watcher.py:__init__():211] watching files in: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files
2024-09-03 14:17:34,746 INFO    SenderThread:59848 [sender.py:_start_run_threads():1124] run started: wo08hjyp with start time 1725365485.044428
2024-09-03 14:17:34,751 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: summary_record
2024-09-03 14:17:34,752 INFO    SenderThread:59848 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-09-03 14:17:34,761 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: check_version
2024-09-03 14:17:34,761 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: check_version
2024-09-03 14:17:34,854 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: run_start
2024-09-03 14:17:34,861 DEBUG   HandlerThread:59848 [system_info.py:__init__():26] System info init
2024-09-03 14:17:34,861 DEBUG   HandlerThread:59848 [system_info.py:__init__():41] System info init done
2024-09-03 14:17:34,861 INFO    HandlerThread:59848 [system_monitor.py:start():194] Starting system monitor
2024-09-03 14:17:34,862 INFO    SystemMonitor:59848 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-09-03 14:17:34,863 INFO    SystemMonitor:59848 [interfaces.py:start():190] Started cpu monitoring
2024-09-03 14:17:34,863 INFO    SystemMonitor:59848 [interfaces.py:start():190] Started disk monitoring
2024-09-03 14:17:34,865 INFO    SystemMonitor:59848 [interfaces.py:start():190] Started gpuapple monitoring
2024-09-03 14:17:34,866 INFO    SystemMonitor:59848 [interfaces.py:start():190] Started memory monitoring
2024-09-03 14:17:34,867 INFO    SystemMonitor:59848 [interfaces.py:start():190] Started network monitoring
2024-09-03 14:17:35,358 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: python_packages
2024-09-03 14:17:35,359 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: python_packages
2024-09-03 14:17:35,359 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:17:35,360 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:17:35,361 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:17:35,589 DEBUG   SenderThread:59848 [sender.py:send():379] send: telemetry
2024-09-03 14:17:35,751 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/requirements.txt
2024-09-03 14:17:35,751 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/output.log
2024-09-03 14:17:35,751 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_created():271] file/dir created: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/wandb-summary.json
2024-09-03 14:17:37,756 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/output.log
2024-09-03 14:17:39,605 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:17:43,772 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/output.log
2024-09-03 14:17:45,541 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:17:50,363 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:17:50,365 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:17:50,367 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:17:50,630 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:17:55,652 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:00,665 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:05,367 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:18:05,368 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:18:05,369 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:18:06,600 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:07,876 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/config.yaml
2024-09-03 14:18:12,053 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:17,075 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:20,371 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:18:20,372 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:18:20,373 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:18:22,615 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:27,637 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:32,653 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:34,877 DEBUG   SystemMonitor:59848 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-09-03 14:18:34,879 DEBUG   SenderThread:59848 [sender.py:send():379] send: telemetry
2024-09-03 14:18:34,881 DEBUG   SenderThread:59848 [sender.py:send():379] send: stats
2024-09-03 14:18:35,376 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:18:35,377 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:18:35,377 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:18:38,624 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:38,992 INFO    Thread-16 :59848 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/siemsj/projects/xlstm/experiments/wandb/run-20240903_141734-wo08hjyp/files/config.yaml
2024-09-03 14:18:43,840 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:48,862 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:50,382 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:18:50,383 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:18:50,383 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:18:54,712 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:18:59,729 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:04,750 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:04,892 DEBUG   SenderThread:59848 [sender.py:send():379] send: stats
2024-09-03 14:19:05,382 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:19:05,382 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:19:05,387 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:19:10,620 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:15,646 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:20,387 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:19:20,388 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:19:20,389 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:19:20,669 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:25,690 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:30,703 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:34,900 DEBUG   SenderThread:59848 [sender.py:send():379] send: stats
2024-09-03 14:19:35,391 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: internal_messages
2024-09-03 14:19:35,392 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: stop_status
2024-09-03 14:19:35,393 DEBUG   SenderThread:59848 [sender.py:send_request():406] send_request: stop_status
2024-09-03 14:19:36,590 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
2024-09-03 14:19:41,611 DEBUG   HandlerThread:59848 [handler.py:handle_request():146] handle_request: status_report
