FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  885425         	|  3442.7         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  164.22         	|  20             	|  3284.5         	|  95.405         	|
|  run_training_batch                                                                                                                                                                                         	|  0.1387         	|  20000          	|  2774.0         	|  80.576         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.13829        	|  20000          	|  2765.9         	|  80.34          	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.032671       	|  20000          	|  653.42         	|  18.98          	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  1.0024         	|  310            	|  310.74         	|  9.0262         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0081855      	|  20000          	|  163.71         	|  4.7553         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  145.37         	|  1              	|  145.37         	|  4.2226         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0046612      	|  20000          	|  93.224         	|  2.7079         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00064931     	|  20000          	|  12.986         	|  0.37721        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.20952        	|  20             	|  4.1905         	|  0.12172        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00018713     	|  20310          	|  3.8007         	|  0.1104         	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00011416     	|  20310          	|  2.3186         	|  0.067349       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0028442      	|  310            	|  0.88171        	|  0.025611       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.0702e-05     	|  20000          	|  0.61405        	|  0.017836       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.1151e-05     	|  20000          	|  0.42302        	|  0.012288       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.9258e-05     	|  20000          	|  0.38517        	|  0.011188       	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.0004509      	|  310            	|  0.13978        	|  0.0040602      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.2486e-06     	|  20000          	|  0.12497        	|  0.0036301      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.6359e-06     	|  20000          	|  0.11272        	|  0.0032741      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  5.6233e-06     	|  20000          	|  0.11247        	|  0.0032668      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  5.3749e-06     	|  20000          	|  0.1075         	|  0.0031225      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.051083       	|  2              	|  0.10217        	|  0.0029676      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.8547e-06     	|  20000          	|  0.097094       	|  0.0028203      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.8182e-06     	|  20000          	|  0.096363       	|  0.0027991      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.7901e-06     	|  20000          	|  0.095802       	|  0.0027828      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  4.6863e-06     	|  20310          	|  0.09518        	|  0.0027647      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.3179e-06     	|  20000          	|  0.086359       	|  0.0025085      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  4.1344e-06     	|  20000          	|  0.082689       	|  0.0024019      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  4.0638e-06     	|  20000          	|  0.081276       	|  0.0023608      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.034e-06      	|  20000          	|  0.080681       	|  0.0023435      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  3.998e-06      	|  20000          	|  0.079961       	|  0.0023226      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.9968e-06     	|  20000          	|  0.079935       	|  0.0023219      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  3.9764e-06     	|  20000          	|  0.079529       	|  0.0023101      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  3.9683e-06     	|  20000          	|  0.079366       	|  0.0023054      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  3.9568e-06     	|  20000          	|  0.079135       	|  0.0022987      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  3.9373e-06     	|  20000          	|  0.078746       	|  0.0022874      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  3.9167e-06     	|  20000          	|  0.078335       	|  0.0022754      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  3.9153e-06     	|  20000          	|  0.078306       	|  0.0022746      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  3.9039e-06     	|  20000          	|  0.078078       	|  0.002268       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  3.9026e-06     	|  20000          	|  0.078052       	|  0.0022672      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.8713e-06     	|  20000          	|  0.077427       	|  0.002249       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  3.2111e-06     	|  20310          	|  0.065217       	|  0.0018944      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  3.0879e-06     	|  20000          	|  0.061757       	|  0.0017939      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  2.8411e-06     	|  20000          	|  0.056823       	|  0.0016505      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  2.7947e-06     	|  20000          	|  0.055893       	|  0.0016235      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.7538e-06     	|  20000          	|  0.055077       	|  0.0015998      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  2.7036e-06     	|  20000          	|  0.054072       	|  0.0015706      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  2.5991e-06     	|  20000          	|  0.051983       	|  0.0015099      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  2.5336e-06     	|  20000          	|  0.050671       	|  0.0014719      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  2.5203e-06     	|  20000          	|  0.050406       	|  0.0014642      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  2.1761e-06     	|  20000          	|  0.043523       	|  0.0012642      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0041629      	|  2              	|  0.0083258      	|  0.00024184     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00039401     	|  20             	|  0.0078803      	|  0.0002289      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0071117      	|  1              	|  0.0071117      	|  0.00020657     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.0002926      	|  20             	|  0.005852       	|  0.00016998     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  0.0052624      	|  1              	|  0.0052624      	|  0.00015286     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  1.6868e-05     	|  310            	|  0.005229       	|  0.00015189     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00012641     	|  20             	|  0.0025282      	|  7.3437e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  8.051e-06      	|  310            	|  0.0024958      	|  7.2496e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  5.6954e-06     	|  310            	|  0.0017656      	|  5.1285e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  5.5388e-06     	|  310            	|  0.001717       	|  4.9875e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  4.8208e-06     	|  310            	|  0.0014944      	|  4.3409e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  4.6336e-06     	|  310            	|  0.0014364      	|  4.1723e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00070252     	|  2              	|  0.001405       	|  4.0813e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00067801     	|  2              	|  0.001356       	|  3.9388e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.0012566      	|  1              	|  0.0012566      	|  3.6499e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  3.6241e-06     	|  310            	|  0.0011235      	|  3.2633e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  3.5994e-06     	|  310            	|  0.0011158      	|  3.2411e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  2.7138e-06     	|  310            	|  0.00084129     	|  2.4437e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00041583     	|  2              	|  0.00083166     	|  2.4157e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00038416     	|  2              	|  0.00076831     	|  2.2317e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00052373     	|  1              	|  0.00052373     	|  1.5213e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00032266     	|  1              	|  0.00032266     	|  9.3724e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  1.3193e-05     	|  20             	|  0.00026386     	|  7.6643e-06     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00026211     	|  1              	|  0.00026211     	|  7.6135e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  7.8674e-06     	|  20             	|  0.00015735     	|  4.5705e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  5.4857e-06     	|  22             	|  0.00012069     	|  3.5056e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  5.6535e-05     	|  2              	|  0.00011307     	|  3.2844e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.3743e-06     	|  22             	|  9.6235e-05     	|  2.7954e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  9.61e-05       	|  1              	|  9.61e-05       	|  2.7914e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  4.0634e-06     	|  22             	|  8.9396e-05     	|  2.5967e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  3.9556e-06     	|  22             	|  8.7024e-05     	|  2.5278e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  4.33e-06       	|  20             	|  8.6601e-05     	|  2.5155e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  3.9331e-06     	|  22             	|  8.6529e-05     	|  2.5134e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  3.8891e-06     	|  22             	|  8.556e-05      	|  2.4853e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  3.4252e-06     	|  22             	|  7.5354e-05     	|  2.1888e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  3.562e-06      	|  20             	|  7.1241e-05     	|  2.0693e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  2.7208e-06     	|  22             	|  5.9857e-05     	|  1.7387e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  2.8179e-06     	|  20             	|  5.6359e-05     	|  1.6371e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.8019e-05     	|  2              	|  3.6038e-05     	|  1.0468e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  3.3594e-05     	|  1              	|  3.3594e-05     	|  9.758e-07      	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.5644e-05     	|  2              	|  3.1288e-05     	|  9.0882e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  9.0091e-06     	|  2              	|  1.8018e-05     	|  5.2338e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  7.2983e-06     	|  2              	|  1.4597e-05     	|  4.2399e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  1.327e-05      	|  1              	|  1.327e-05      	|  3.8547e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  1.2291e-05     	|  1              	|  1.2291e-05     	|  3.5701e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  5.9721e-06     	|  2              	|  1.1944e-05     	|  3.4695e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  1.1314e-05     	|  1              	|  1.1314e-05     	|  3.2863e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  9.4976e-06     	|  1              	|  9.4976e-06     	|  2.7588e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  8.5207e-06     	|  1              	|  8.5207e-06     	|  2.475e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  4.0508e-06     	|  2              	|  8.1016e-06     	|  2.3533e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  8.0317e-06     	|  1              	|  8.0317e-06     	|  2.333e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  4.0159e-06     	|  2              	|  8.0317e-06     	|  2.333e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  3.9805e-06     	|  2              	|  7.9609e-06     	|  2.3124e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  7.6126e-06     	|  1              	|  7.6126e-06     	|  2.2113e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  3.7714e-06     	|  2              	|  7.5428e-06     	|  2.191e-07      	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  7.5419e-06     	|  1              	|  7.5419e-06     	|  2.1907e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  3.7365e-06     	|  2              	|  7.4729e-06     	|  2.1707e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.736e-06      	|  2              	|  7.472e-06      	|  2.1704e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  7.0538e-06     	|  1              	|  7.0538e-06     	|  2.0489e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.5269e-06     	|  2              	|  7.0538e-06     	|  2.0489e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  3.5269e-06     	|  2              	|  7.0538e-06     	|  2.0489e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  6.1449e-06     	|  1              	|  6.1449e-06     	|  1.7849e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  6.0759e-06     	|  1              	|  6.0759e-06     	|  1.7649e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  6.0759e-06     	|  1              	|  6.0759e-06     	|  1.7649e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.6432e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.6432e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.6432e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.6432e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  5.587e-06      	|  1              	|  5.587e-06      	|  1.6229e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  5.2378e-06     	|  1              	|  5.2378e-06     	|  1.5214e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.2378e-06     	|  1              	|  5.2378e-06     	|  1.5214e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  4.2608e-06     	|  1              	|  4.2608e-06     	|  1.2376e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  4.19e-06       	|  1              	|  4.19e-06       	|  1.2171e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  4.19e-06       	|  1              	|  4.19e-06       	|  1.2171e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  1.1159e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  1.1159e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  1.1159e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  1.1156e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  1.1156e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  1.1156e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  1.1156e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  1.1156e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  1.0956e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  1.0956e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.8859e-06     	|  2              	|  3.7719e-06     	|  1.0956e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  1.0953e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  3.702e-06      	|  1              	|  3.702e-06      	|  1.0753e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  3.702e-06      	|  1              	|  3.702e-06      	|  1.0753e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  3.7011e-06     	|  1              	|  3.7011e-06     	|  1.0751e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.8859e-06     	|  1              	|  1.8859e-06     	|  5.4781e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.4678e-06     	|  1              	|  1.4678e-06     	|  4.2634e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
