FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  880977         	|  2796.7         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  138.76         	|  20             	|  2775.3         	|  99.232         	|
|  run_training_batch                                                                                                                                                                                         	|  0.12771        	|  20000          	|  2554.2         	|  91.329         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.12726        	|  20000          	|  2545.2         	|  91.007         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.034647       	|  20000          	|  692.94         	|  24.777         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0056154      	|  20000          	|  112.31         	|  4.0157         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0049156      	|  20000          	|  98.311         	|  3.5152         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  2.4166         	|  32             	|  77.33          	|  2.765          	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00056274     	|  20000          	|  11.255         	|  0.40243        	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  8.5443         	|  1              	|  8.5443         	|  0.30551        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.26007        	|  20             	|  5.2014         	|  0.18598        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00019954     	|  20032          	|  3.9973         	|  0.14293        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00012755     	|  20032          	|  2.555          	|  0.091358       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.5817e-05     	|  20000          	|  0.71633        	|  0.025613       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.017456       	|  32             	|  0.55858        	|  0.019973       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.0636e-05     	|  20000          	|  0.41272        	|  0.014757       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.0136e-05     	|  20000          	|  0.40272        	|  0.0144         	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.059571       	|  2              	|  0.11914        	|  0.0042601      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.41e-06       	|  20000          	|  0.1082         	|  0.0038688      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  5.3868e-06     	|  20000          	|  0.10774        	|  0.0038522      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.9909e-06     	|  20000          	|  0.079817       	|  0.002854       	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.8337e-06     	|  20000          	|  0.076675       	|  0.0027416      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.8168e-06     	|  20000          	|  0.076335       	|  0.0027294      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.4741e-06     	|  20000          	|  0.069481       	|  0.0024844      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.2591e-06     	|  20000          	|  0.065183       	|  0.0023307      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.1605e-06     	|  20000          	|  0.063211       	|  0.0022602      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.0011e-06     	|  20000          	|  0.060022       	|  0.0021462      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.792e-06      	|  20000          	|  0.05584        	|  0.0019966      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.606e-06      	|  20000          	|  0.05212        	|  0.0018636      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.5665e-06     	|  20000          	|  0.051331       	|  0.0018354      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.514e-06      	|  20000          	|  0.05028        	|  0.0017978      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.4053e-06     	|  20000          	|  0.048105       	|  0.00172        	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.3957e-06     	|  20000          	|  0.047915       	|  0.0017132      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  2.3725e-06     	|  20032          	|  0.047527       	|  0.0016994      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.3017e-06     	|  20000          	|  0.046035       	|  0.001646       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.2889e-06     	|  20000          	|  0.045779       	|  0.0016369      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.2664e-06     	|  20000          	|  0.045328       	|  0.0016207      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.1265e-06     	|  20000          	|  0.042531       	|  0.0015207      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.1174e-06     	|  20000          	|  0.042347       	|  0.0015142      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.1e-06        	|  20000          	|  0.042          	|  0.0015018      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.0994e-06     	|  20000          	|  0.041988       	|  0.0015013      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  2.0048e-06     	|  20000          	|  0.040096       	|  0.0014337      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.4558e-06     	|  20032          	|  0.029163       	|  0.0010427      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.4451e-06     	|  20000          	|  0.028901       	|  0.0010334      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.4207e-06     	|  20000          	|  0.028414       	|  0.001016       	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.3674e-06     	|  20000          	|  0.027348       	|  0.00097785     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.3609e-06     	|  20000          	|  0.027218       	|  0.00097322     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.316e-06      	|  20000          	|  0.026319       	|  0.00094108     	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.2251e-06     	|  20000          	|  0.024502       	|  0.00087608     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1629e-06     	|  20000          	|  0.023259       	|  0.00083164     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.1039e-06     	|  20000          	|  0.022079       	|  0.00078944     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.016421       	|  1              	|  0.016421       	|  0.00058715     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00048485     	|  32             	|  0.015515       	|  0.00055476     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0053784      	|  2              	|  0.010757       	|  0.00038462     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00033303     	|  20             	|  0.0066606      	|  0.00023816     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00024163     	|  20             	|  0.0048327      	|  0.0001728      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  7.9632e-05     	|  32             	|  0.0025482      	|  9.1114e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00011828     	|  20             	|  0.0023656      	|  8.4584e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00050409     	|  2              	|  0.0010082      	|  3.6049e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00038734     	|  2              	|  0.00077467     	|  2.7699e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00037632     	|  2              	|  0.00075264     	|  2.6912e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00031633     	|  2              	|  0.00063265     	|  2.2621e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00051772     	|  1              	|  0.00051772     	|  1.8511e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00029571     	|  1              	|  0.00029571     	|  1.0573e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00027511     	|  1              	|  0.00027511     	|  9.8368e-06     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00026717     	|  1              	|  0.00026717     	|  9.5529e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  8.0024e-06     	|  32             	|  0.00025608     	|  9.1563e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.1622e-06     	|  32             	|  0.00013319     	|  4.7624e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  6.3679e-05     	|  2              	|  0.00012736     	|  4.5538e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.2524e-06     	|  20             	|  0.00012505     	|  4.4712e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.4315e-06     	|  32             	|  0.00010981     	|  3.9263e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.376e-06      	|  32             	|  0.00010803     	|  3.8628e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.162e-06      	|  20             	|  0.00010324     	|  3.6915e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.0917e-06     	|  32             	|  9.8933e-05     	|  3.5375e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.4109e-06     	|  22             	|  7.504e-05      	|  2.6831e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.3382e-06     	|  32             	|  7.4822e-05     	|  2.6754e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.0586e-06     	|  22             	|  6.729e-05      	|  2.406e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.032e-06      	|  20             	|  6.064e-05      	|  2.1683e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  5.421e-05      	|  1              	|  5.421e-05      	|  1.9384e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.6139e-06     	|  32             	|  5.1646e-05     	|  1.8466e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.2913e-06     	|  22             	|  5.0409e-05     	|  1.8024e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.246e-06      	|  22             	|  4.9412e-05     	|  1.7668e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.5187e-06     	|  32             	|  4.8597e-05     	|  1.7377e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.1961e-06     	|  22             	|  4.8314e-05     	|  1.7275e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.189e-06      	|  22             	|  4.8159e-05     	|  1.722e-06      	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.5236e-06     	|  20             	|  3.0472e-05     	|  1.0896e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3699e-06     	|  22             	|  3.0138e-05     	|  1.0776e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.3463e-06     	|  22             	|  2.9618e-05     	|  1.059e-06      	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.3796e-06     	|  20             	|  2.7591e-05     	|  9.8656e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  9.1097e-06     	|  2              	|  1.8219e-05     	|  6.5146e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  7.0347e-06     	|  2              	|  1.4069e-05     	|  5.0307e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.085e-06      	|  2              	|  1.017e-05      	|  3.6364e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.3998e-06     	|  1              	|  9.3998e-06     	|  3.361e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.5053e-06     	|  2              	|  9.0105e-06     	|  3.2218e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  4.3893e-06     	|  2              	|  8.7786e-06     	|  3.1389e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.41e-06       	|  2              	|  6.8201e-06     	|  2.4386e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  6.7493e-06     	|  1              	|  6.7493e-06     	|  2.4133e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  6.0396e-06     	|  1              	|  6.0396e-06     	|  2.1595e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.7497e-06     	|  2              	|  5.4995e-06     	|  1.9664e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.725e-06      	|  2              	|  5.4501e-06     	|  1.9487e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  5.0301e-06     	|  1              	|  5.0301e-06     	|  1.7986e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.4196e-06     	|  2              	|  4.8392e-06     	|  1.7303e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.5793e-06     	|  1              	|  4.5793e-06     	|  1.6374e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.5197e-06     	|  1              	|  4.5197e-06     	|  1.6161e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.2301e-06     	|  2              	|  4.4601e-06     	|  1.5948e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.1998e-06     	|  2              	|  4.3996e-06     	|  1.5731e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  4.2999e-06     	|  1              	|  4.2999e-06     	|  1.5375e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.2692e-06     	|  1              	|  4.2692e-06     	|  1.5265e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.0401e-06     	|  2              	|  4.0801e-06     	|  1.4589e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.9451e-06     	|  2              	|  3.8901e-06     	|  1.391e-07      	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  3.8696e-06     	|  1              	|  3.8696e-06     	|  1.3836e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.7802e-06     	|  1              	|  3.7802e-06     	|  1.3517e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.4701e-06     	|  1              	|  3.4701e-06     	|  1.2408e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.3798e-06     	|  1              	|  3.3798e-06     	|  1.2085e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.2093e-06     	|  1              	|  3.2093e-06     	|  1.1475e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.9402e-06     	|  1              	|  2.9402e-06     	|  1.0513e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.8806e-06     	|  1              	|  2.8806e-06     	|  1.03e-07       	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.8396e-06     	|  1              	|  2.8396e-06     	|  1.0153e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.6897e-06     	|  1              	|  2.6897e-06     	|  9.6172e-08     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.6496e-06     	|  1              	|  2.6496e-06     	|  9.474e-08      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.6403e-06     	|  1              	|  2.6403e-06     	|  9.4407e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.6096e-06     	|  1              	|  2.6096e-06     	|  9.3308e-08     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.6003e-06     	|  1              	|  2.6003e-06     	|  9.2975e-08     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.55e-06       	|  1              	|  2.55e-06       	|  9.1177e-08     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.5202e-06     	|  1              	|  2.5202e-06     	|  9.0111e-08     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.2098e-06     	|  2              	|  2.4196e-06     	|  8.6515e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.4e-06        	|  1              	|  2.4e-06        	|  8.5815e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.3106e-06     	|  1              	|  2.3106e-06     	|  8.2618e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.2799e-06     	|  1              	|  2.2799e-06     	|  8.1519e-08     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.2203e-06     	|  1              	|  2.2203e-06     	|  7.9388e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1504e-06     	|  1              	|  2.1504e-06     	|  7.6891e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9595e-06     	|  1              	|  1.9595e-06     	|  7.0064e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9493e-06     	|  1              	|  1.9493e-06     	|  6.9698e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.79e-06       	|  1              	|  1.79e-06       	|  6.4003e-08     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.7099e-06     	|  1              	|  1.7099e-06     	|  6.114e-08      	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.6801e-06     	|  1              	|  1.6801e-06     	|  6.0074e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.6699e-06     	|  1              	|  1.6699e-06     	|  5.9708e-08     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.4799e-06     	|  1              	|  1.4799e-06     	|  5.2914e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.4398e-06     	|  1              	|  1.4398e-06     	|  5.1482e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.3504e-06     	|  1              	|  1.3504e-06     	|  4.8286e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.22e-06       	|  1              	|  1.22e-06       	|  4.3624e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.2005e-06     	|  1              	|  1.2005e-06     	|  4.2924e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.0105e-06     	|  1              	|  1.0105e-06     	|  3.6131e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
