FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881265         	|  3791.5         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  187.21         	|  20             	|  3744.2         	|  98.752         	|
|  run_training_batch                                                                                                                                                                                         	|  0.18089        	|  20000          	|  3617.7         	|  95.417         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.18052        	|  20000          	|  3610.5         	|  95.226         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.049499       	|  20000          	|  989.99         	|  26.111         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0048188      	|  20000          	|  96.376         	|  2.5419         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0036346      	|  20000          	|  72.692         	|  1.9172         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  34.555         	|  1              	|  34.555         	|  0.91139        	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.56324        	|  50             	|  28.162         	|  0.74277        	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00049222     	|  20000          	|  9.8444         	|  0.25965        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.2679         	|  20             	|  5.3581         	|  0.14132        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00013701     	|  20050          	|  2.7471         	|  0.072455       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  8.7214e-05     	|  20050          	|  1.7486         	|  0.046121       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.7327e-05     	|  20000          	|  0.54654        	|  0.014415       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.6638e-05     	|  20000          	|  0.33276        	|  0.0087767      	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.5375e-05     	|  20000          	|  0.3075         	|  0.0081103      	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0038648      	|  50             	|  0.19324        	|  0.0050967      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.059471       	|  2              	|  0.11894        	|  0.0031371      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.6356e-06     	|  20000          	|  0.092712       	|  0.0024453      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.7458e-06     	|  20000          	|  0.074917       	|  0.0019759      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.2151e-06     	|  20000          	|  0.064303       	|  0.001696       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  2.9885e-06     	|  20000          	|  0.059769       	|  0.0015764      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.9508e-06     	|  20000          	|  0.059015       	|  0.0015565      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.9402e-06     	|  20000          	|  0.058805       	|  0.001551       	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.9272e-06     	|  20000          	|  0.058543       	|  0.0015441      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.6588e-06     	|  20000          	|  0.053176       	|  0.0014025      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.6555e-06     	|  20000          	|  0.05311        	|  0.0014008      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.5471e-06     	|  20000          	|  0.050942       	|  0.0013436      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.5028e-06     	|  20000          	|  0.050057       	|  0.0013202      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.4945e-06     	|  20000          	|  0.04989        	|  0.0013158      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.4773e-06     	|  20000          	|  0.049545       	|  0.0013068      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.374e-06      	|  20000          	|  0.047481       	|  0.0012523      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.3357e-06     	|  20000          	|  0.046714       	|  0.0012321      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.3318e-06     	|  20000          	|  0.046635       	|  0.00123        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.2532e-06     	|  20000          	|  0.045065       	|  0.0011886      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.1207e-06     	|  20000          	|  0.042415       	|  0.0011187      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.1017e-06     	|  20000          	|  0.042034       	|  0.0011086      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  1.9888e-06     	|  20000          	|  0.039777       	|  0.0010491      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.9714e-06     	|  20000          	|  0.039428       	|  0.0010399      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.9421e-06     	|  20000          	|  0.038842       	|  0.0010245      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.3674e-06     	|  20050          	|  0.027415       	|  0.00072308     	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.2403e-06     	|  20000          	|  0.024806       	|  0.00065425     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.1731e-06     	|  20000          	|  0.023463       	|  0.00061883     	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1597e-06     	|  20050          	|  0.023252       	|  0.00061328     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.1027e-06     	|  20000          	|  0.022054       	|  0.00058167     	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.0523e-06     	|  20000          	|  0.021046       	|  0.0005551      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.0361e-06     	|  20000          	|  0.020723       	|  0.00054657     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.0107e-06     	|  20000          	|  0.020215       	|  0.00053317     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.0102e-06     	|  20000          	|  0.020203       	|  0.00053286     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  9.8822e-07     	|  20000          	|  0.019764       	|  0.00052129     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00039132     	|  50             	|  0.019566       	|  0.00051605     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  9.4502e-07     	|  20000          	|  0.0189         	|  0.0004985      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0049675      	|  2              	|  0.0099351      	|  0.00026204     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00025449     	|  20             	|  0.0050897      	|  0.00013424     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00022603     	|  20             	|  0.0045205      	|  0.00011923     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  6.5687e-05     	|  50             	|  0.0032844      	|  8.6625e-05     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.002366       	|  1              	|  0.002366       	|  6.2404e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010396     	|  20             	|  0.0020792      	|  5.4838e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00087459     	|  2              	|  0.0017492      	|  4.6135e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00054532     	|  2              	|  0.0010906      	|  2.8766e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00048008     	|  2              	|  0.00096016     	|  2.5324e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00044351     	|  2              	|  0.00088702     	|  2.3395e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00065338     	|  1              	|  0.00065338     	|  1.7233e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00040171     	|  1              	|  0.00040171     	|  1.0595e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00035694     	|  1              	|  0.00035694     	|  9.4142e-06     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00028515     	|  1              	|  0.00028515     	|  7.5208e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  5.6645e-06     	|  50             	|  0.00028322     	|  7.4701e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.584e-06      	|  50             	|  0.0001792      	|  4.7264e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.4609e-06     	|  50             	|  0.00017305     	|  4.5641e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.1354e-06     	|  50             	|  0.00015677     	|  4.1348e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  2.3048e-06     	|  50             	|  0.00011524     	|  3.0395e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.7048e-06     	|  20             	|  0.0001141      	|  3.0093e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.6299e-06     	|  20             	|  9.2599e-05     	|  2.4423e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.7515e-06     	|  50             	|  8.7575e-05     	|  2.3098e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.8454e-05     	|  2              	|  7.6909e-05     	|  2.0285e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.4458e-06     	|  50             	|  7.2288e-05     	|  1.9066e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.1663e-06     	|  22             	|  6.9659e-05     	|  1.8373e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.8485e-06     	|  22             	|  6.2667e-05     	|  1.6528e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.9533e-06     	|  20             	|  5.9065e-05     	|  1.5579e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.4164e-06     	|  22             	|  5.3162e-05     	|  1.4021e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.2849e-06     	|  22             	|  5.0268e-05     	|  1.3258e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  9.9944e-07     	|  50             	|  4.9972e-05     	|  1.318e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.4559e-05     	|  1              	|  4.4559e-05     	|  1.1753e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  1.9981e-06     	|  22             	|  4.3958e-05     	|  1.1594e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  1.9576e-06     	|  22             	|  4.3067e-05     	|  1.1359e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.3578e-06     	|  22             	|  2.9871e-05     	|  7.8786e-07     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.2449e-06     	|  22             	|  2.7388e-05     	|  7.2237e-07     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.351e-06      	|  20             	|  2.702e-05      	|  7.1267e-07     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.2471e-06     	|  20             	|  2.4943e-05     	|  6.5787e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.164e-05      	|  2              	|  2.328e-05      	|  6.1402e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  6.835e-06      	|  2              	|  1.367e-05      	|  3.6055e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.8052e-06     	|  2              	|  9.6103e-06     	|  2.5347e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  9.5796e-06     	|  1              	|  9.5796e-06     	|  2.5266e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  4.6846e-06     	|  2              	|  9.3691e-06     	|  2.4711e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  8.7302e-06     	|  1              	|  8.7302e-06     	|  2.3026e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  8.6986e-06     	|  1              	|  8.6986e-06     	|  2.2942e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  4.2049e-06     	|  2              	|  8.4098e-06     	|  2.2181e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.0997e-06     	|  2              	|  8.1994e-06     	|  2.1626e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  7.31e-06       	|  1              	|  7.31e-06       	|  1.928e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  3.54e-06       	|  2              	|  7.0799e-06     	|  1.8673e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.9551e-06     	|  2              	|  5.9102e-06     	|  1.5588e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  5.6699e-06     	|  1              	|  5.6699e-06     	|  1.4954e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.7902e-06     	|  2              	|  5.5805e-06     	|  1.4719e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.4501e-06     	|  1              	|  5.4501e-06     	|  1.4375e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.3504e-06     	|  1              	|  5.3504e-06     	|  1.4112e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  5.0301e-06     	|  1              	|  5.0301e-06     	|  1.3267e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.8801e-06     	|  1              	|  4.8801e-06     	|  1.2871e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.7199e-06     	|  1              	|  4.7199e-06     	|  1.2449e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.3353e-06     	|  2              	|  4.6706e-06     	|  1.2319e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.6603e-06     	|  1              	|  4.6603e-06     	|  1.2292e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.5104e-06     	|  1              	|  4.5104e-06     	|  1.1896e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.4499e-06     	|  1              	|  4.4499e-06     	|  1.1737e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.1798e-06     	|  2              	|  4.3595e-06     	|  1.1498e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  4.3204e-06     	|  1              	|  4.3204e-06     	|  1.1395e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.8901e-06     	|  1              	|  3.8901e-06     	|  1.026e-07      	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  3.77e-06       	|  1              	|  3.77e-06       	|  9.9434e-08     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.3397e-06     	|  1              	|  3.3397e-06     	|  8.8085e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  3.3397e-06     	|  1              	|  3.3397e-06     	|  8.8085e-08     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.2103e-06     	|  1              	|  3.2103e-06     	|  8.4671e-08     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.01e-06       	|  1              	|  3.01e-06       	|  7.939e-08      	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.4855e-06     	|  2              	|  2.9709e-06     	|  7.8358e-08     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  2.9095e-06     	|  1              	|  2.9095e-06     	|  7.6737e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.8396e-06     	|  1              	|  2.8396e-06     	|  7.4895e-08     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.4049e-06     	|  2              	|  2.8098e-06     	|  7.4109e-08     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.7101e-06     	|  1              	|  2.7101e-06     	|  7.148e-08      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.6096e-06     	|  1              	|  2.6096e-06     	|  6.8827e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.5397e-06     	|  1              	|  2.5397e-06     	|  6.6985e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.5099e-06     	|  1              	|  2.5099e-06     	|  6.6199e-08     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.2452e-06     	|  2              	|  2.4904e-06     	|  6.5683e-08     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.4503e-06     	|  1              	|  2.4503e-06     	|  6.4627e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  2.4401e-06     	|  1              	|  2.4401e-06     	|  6.4357e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.3199e-06     	|  1              	|  2.3199e-06     	|  6.1188e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.2603e-06     	|  1              	|  2.2603e-06     	|  5.9616e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.1597e-06     	|  1              	|  2.1597e-06     	|  5.6963e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  2.0806e-06     	|  1              	|  2.0806e-06     	|  5.4875e-08     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.0694e-06     	|  1              	|  2.0694e-06     	|  5.4581e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9297e-06     	|  1              	|  1.9297e-06     	|  5.0896e-08     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.9204e-06     	|  1              	|  1.9204e-06     	|  5.065e-08      	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.7295e-06     	|  1              	|  1.7295e-06     	|  4.5615e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.7202e-06     	|  1              	|  1.7202e-06     	|  4.5369e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.4799e-06     	|  1              	|  1.4799e-06     	|  3.9032e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.2806e-06     	|  1              	|  1.2806e-06     	|  3.3775e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  9.4064e-07     	|  1              	|  9.4064e-07     	|  2.4809e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
