FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881393         	|  2692.8         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  132.29         	|  20             	|  2645.8         	|  98.255         	|
|  run_training_batch                                                                                                                                                                                         	|  0.12717        	|  20000          	|  2543.5         	|  94.456         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.12685        	|  20000          	|  2537.0         	|  94.215         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.034606       	|  20000          	|  692.13         	|  25.703         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.004673       	|  20000          	|  93.46          	|  3.4708         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0022215      	|  20000          	|  44.43          	|  1.65           	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  36.526         	|  1              	|  36.526         	|  1.3564         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.61535        	|  58             	|  35.69          	|  1.3254         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00044401     	|  20000          	|  8.8802         	|  0.32978        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.22316        	|  20             	|  4.4632         	|  0.16575        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00011598     	|  20058          	|  2.3264         	|  0.086394       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  7.399e-05      	|  20058          	|  1.4841         	|  0.055114       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.3756e-05     	|  20000          	|  0.47513        	|  0.017644       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0059822      	|  58             	|  0.34697        	|  0.012885       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.4363e-05     	|  20000          	|  0.28726        	|  0.010668       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.388e-05      	|  20000          	|  0.27759        	|  0.010309       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.068703       	|  2              	|  0.13741        	|  0.0051027      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.2874e-06     	|  20000          	|  0.085749       	|  0.0031844      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.4833e-06     	|  20000          	|  0.069667       	|  0.0025872      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.0101e-06     	|  20000          	|  0.060202       	|  0.0022357      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  2.9777e-06     	|  20000          	|  0.059554       	|  0.0022116      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.9349e-06     	|  20000          	|  0.058698       	|  0.0021798      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.8377e-06     	|  20000          	|  0.056753       	|  0.0021076      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.4867e-06     	|  20000          	|  0.049735       	|  0.001847       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.4272e-06     	|  20000          	|  0.048543       	|  0.0018027      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.316e-06      	|  20000          	|  0.046319       	|  0.0017201      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3118e-06     	|  20000          	|  0.046237       	|  0.0017171      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.1857e-06     	|  20000          	|  0.043714       	|  0.0016234      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.1848e-06     	|  20000          	|  0.043696       	|  0.0016227      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.1578e-06     	|  20000          	|  0.043156       	|  0.0016027      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.0615e-06     	|  20000          	|  0.041231       	|  0.0015312      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.0606e-06     	|  20000          	|  0.041211       	|  0.0015304      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.0444e-06     	|  20000          	|  0.040887       	|  0.0015184      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.0199e-06     	|  20000          	|  0.040398       	|  0.0015002      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2e-06          	|  20000          	|  0.04           	|  0.0014855      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  1.9467e-06     	|  20000          	|  0.038933       	|  0.0014458      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.942e-06      	|  20000          	|  0.038841       	|  0.0014424      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.9044e-06     	|  20000          	|  0.038088       	|  0.0014145      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.904e-06      	|  20000          	|  0.038081       	|  0.0014142      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00042842     	|  58             	|  0.024848       	|  0.00092277     	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.1458e-06     	|  20058          	|  0.022983       	|  0.0008535      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.0652e-06     	|  20058          	|  0.021365       	|  0.00079344     	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.0658e-06     	|  20000          	|  0.021316       	|  0.00079158     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.7901e-07     	|  20000          	|  0.01958        	|  0.00072714     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  9.6819e-07     	|  20000          	|  0.019364       	|  0.0007191      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  9.5395e-07     	|  20000          	|  0.019079       	|  0.00070853     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  8.8316e-07     	|  20000          	|  0.017663       	|  0.00065595     	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.7527e-07     	|  20000          	|  0.017505       	|  0.00065008     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.5792e-07     	|  20000          	|  0.017158       	|  0.0006372      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  8.3047e-07     	|  20000          	|  0.016609       	|  0.00061681     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  7.9261e-07     	|  20000          	|  0.015852       	|  0.00058869     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0040908      	|  2              	|  0.0081816      	|  0.00030384     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00025356     	|  20             	|  0.0050713      	|  0.00018833     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00022217     	|  20             	|  0.0044434      	|  0.00016501     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0034273      	|  1              	|  0.0034273      	|  0.00012728     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.4114e-05     	|  58             	|  0.0025586      	|  9.5018e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  9.6518e-05     	|  20             	|  0.0019304      	|  7.1686e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00055396     	|  2              	|  0.0011079      	|  4.1144e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00052361     	|  2              	|  0.0010472      	|  3.889e-05      	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00047654     	|  2              	|  0.00095308     	|  3.5394e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00079409     	|  1              	|  0.00079409     	|  2.949e-05      	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00023294     	|  2              	|  0.00046587     	|  1.7301e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00038949     	|  1              	|  0.00038949     	|  1.4464e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.0426e-06     	|  58             	|  0.00035047     	|  1.3015e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00034976     	|  1              	|  0.00034976     	|  1.2989e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00028514     	|  1              	|  0.00028514     	|  1.0589e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.878e-06      	|  58             	|  0.00022492     	|  8.3529e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.9662e-06     	|  58             	|  0.00017204     	|  6.3889e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.8777e-06     	|  58             	|  0.00016691     	|  6.1983e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  2.2327e-06     	|  58             	|  0.0001295      	|  4.8091e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.5425e-06     	|  20             	|  0.00011085     	|  4.1166e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  5.4045e-05     	|  2              	|  0.00010809     	|  4.0141e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.7894e-06     	|  58             	|  0.00010379     	|  3.8542e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5953e-06     	|  58             	|  9.2528e-05     	|  3.4362e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.4594e-06     	|  20             	|  8.9187e-05     	|  3.3121e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.9136e-06     	|  22             	|  8.6099e-05     	|  3.1974e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.1054e-06     	|  58             	|  6.4113e-05     	|  2.3809e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.7277e-06     	|  22             	|  6.001e-05      	|  2.2285e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.7155e-06     	|  20             	|  5.4309e-05     	|  2.0168e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.0728e-06     	|  22             	|  4.5602e-05     	|  1.6935e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.0051e-06     	|  22             	|  4.4111e-05     	|  1.6381e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  1.9865e-06     	|  22             	|  4.3703e-05     	|  1.623e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  1.9322e-06     	|  22             	|  4.2508e-05     	|  1.5786e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.4544e-06     	|  20             	|  2.9088e-05     	|  1.0802e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  2.8589e-05     	|  1              	|  2.8589e-05     	|  1.0617e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.2731e-06     	|  22             	|  2.8008e-05     	|  1.0401e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.2093e-06     	|  22             	|  2.6604e-05     	|  9.8798e-07     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.233e-06      	|  20             	|  2.466e-05      	|  9.1577e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.161e-05      	|  2              	|  2.322e-05      	|  8.623e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  1.1045e-05     	|  2              	|  2.2089e-05     	|  8.2031e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  8.2101e-06     	|  2              	|  1.642e-05      	|  6.0979e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.035e-05      	|  1              	|  1.035e-05      	|  3.8435e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.026e-05      	|  1              	|  1.026e-05      	|  3.8103e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.8897e-06     	|  1              	|  9.8897e-06     	|  3.6727e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.2701e-06     	|  2              	|  8.5402e-06     	|  3.1715e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  3.7001e-06     	|  2              	|  7.4003e-06     	|  2.7482e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.8303e-06     	|  1              	|  6.8303e-06     	|  2.5365e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  6.2594e-06     	|  1              	|  6.2594e-06     	|  2.3245e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.9605e-06     	|  1              	|  5.9605e-06     	|  2.2135e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.519e-06      	|  1              	|  5.519e-06      	|  2.0496e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.2704e-06     	|  1              	|  5.2704e-06     	|  1.9572e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  5.1688e-06     	|  1              	|  5.1688e-06     	|  1.9195e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  5.1288e-06     	|  1              	|  5.1288e-06     	|  1.9047e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  4.95e-06       	|  1              	|  4.95e-06       	|  1.8382e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.4498e-06     	|  2              	|  4.8997e-06     	|  1.8196e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.6585e-06     	|  1              	|  4.6585e-06     	|  1.73e-07       	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.5896e-06     	|  1              	|  4.5896e-06     	|  1.7044e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.5206e-06     	|  1              	|  4.5206e-06     	|  1.6788e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.5206e-06     	|  1              	|  4.5206e-06     	|  1.6788e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.3605e-06     	|  1              	|  4.3605e-06     	|  1.6193e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.115e-06      	|  2              	|  4.2301e-06     	|  1.5709e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  4.0596e-06     	|  1              	|  4.0596e-06     	|  1.5076e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  1.9455e-06     	|  2              	|  3.8911e-06     	|  1.445e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.9101e-06     	|  2              	|  3.8203e-06     	|  1.4187e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8147e-06     	|  2              	|  3.6294e-06     	|  1.3478e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.6191e-06     	|  1              	|  3.6191e-06     	|  1.344e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.8002e-06     	|  2              	|  3.6005e-06     	|  1.3371e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.2699e-06     	|  1              	|  3.2699e-06     	|  1.2143e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  3.2606e-06     	|  1              	|  3.2606e-06     	|  1.2109e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  3.1898e-06     	|  1              	|  3.1898e-06     	|  1.1846e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.4999e-06     	|  2              	|  2.9998e-06     	|  1.114e-07      	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  2.9095e-06     	|  1              	|  2.9095e-06     	|  1.0805e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.8694e-06     	|  1              	|  2.8694e-06     	|  1.0656e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  2.8601e-06     	|  1              	|  2.8601e-06     	|  1.0621e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.78e-06       	|  1              	|  2.78e-06       	|  1.0324e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.6803e-06     	|  1              	|  2.6803e-06     	|  9.9538e-08     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.6403e-06     	|  1              	|  2.6403e-06     	|  9.8051e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.6105e-06     	|  1              	|  2.6105e-06     	|  9.6945e-08     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.2801e-06     	|  2              	|  2.5602e-06     	|  9.5077e-08     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.4e-06        	|  1              	|  2.4e-06        	|  8.9128e-08     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.1954e-06     	|  2              	|  2.3907e-06     	|  8.8782e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  7.9824e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.1402e-06     	|  1              	|  2.1402e-06     	|  7.9479e-08     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.1001e-06     	|  1              	|  2.1001e-06     	|  7.7991e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1001e-06     	|  1              	|  2.1001e-06     	|  7.7991e-08     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.0303e-06     	|  1              	|  2.0303e-06     	|  7.5397e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.9409e-06     	|  1              	|  1.9409e-06     	|  7.2077e-08     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  1.9399e-06     	|  1              	|  1.9399e-06     	|  7.2043e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.7304e-06     	|  1              	|  1.7304e-06     	|  6.4261e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.7192e-06     	|  1              	|  1.7192e-06     	|  6.3846e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.6103e-06     	|  1              	|  1.6103e-06     	|  5.9799e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.18e-06       	|  1              	|  1.18e-06       	|  4.382e-08      	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
