FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881841         	|  1332.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  64.912         	|  20             	|  1298.2         	|  97.453         	|
|  run_training_batch                                                                                                                                                                                         	|  0.057233       	|  20000          	|  1144.7         	|  85.924         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.056971       	|  20000          	|  1139.4         	|  85.531         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.013027       	|  20000          	|  260.54         	|  19.557         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0039163      	|  20000          	|  78.326         	|  5.8795         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.58847        	|  86             	|  50.609         	|  3.7989         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0020586      	|  20000          	|  41.171         	|  3.0905         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  16.075         	|  1              	|  16.075         	|  1.2067         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00047531     	|  20000          	|  9.5062         	|  0.71359        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.35532        	|  20             	|  7.1065         	|  0.53345        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00013439     	|  20086          	|  2.6994         	|  0.20263        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  8.6148e-05     	|  20086          	|  1.7304         	|  0.12989        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0064122      	|  86             	|  0.55145        	|  0.041394       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.128e-05      	|  20000          	|  0.4256         	|  0.031947       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.4838e-05     	|  20000          	|  0.29676        	|  0.022277       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.481e-05      	|  20000          	|  0.2962         	|  0.022234       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.6327e-06     	|  20000          	|  0.092654       	|  0.006955       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.042932       	|  2              	|  0.085863       	|  0.0064453      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.8257e-06     	|  20000          	|  0.076513       	|  0.0057435      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.7493e-06     	|  20000          	|  0.074985       	|  0.0056288      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.2985e-06     	|  20000          	|  0.065969       	|  0.004952       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.2546e-06     	|  20000          	|  0.065091       	|  0.0048861      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.0069e-06     	|  20000          	|  0.060137       	|  0.0045142      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.8983e-06     	|  20000          	|  0.057967       	|  0.0043513      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.6348e-06     	|  20000          	|  0.052696       	|  0.0039557      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.5194e-06     	|  20000          	|  0.050388       	|  0.0037824      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.4827e-06     	|  20000          	|  0.049655       	|  0.0037273      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.4708e-06     	|  20000          	|  0.049415       	|  0.0037093      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.4688e-06     	|  20000          	|  0.049376       	|  0.0037064      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.4197e-06     	|  20000          	|  0.048394       	|  0.0036327      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.3242e-06     	|  20000          	|  0.046485       	|  0.0034894      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.2961e-06     	|  20000          	|  0.045923       	|  0.0034472      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.2519e-06     	|  20000          	|  0.045038       	|  0.0033808      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.24e-06       	|  20000          	|  0.0448         	|  0.0033629      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.2315e-06     	|  20000          	|  0.044631       	|  0.0033502      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.2178e-06     	|  20000          	|  0.044356       	|  0.0033295      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.1908e-06     	|  20000          	|  0.043816       	|  0.003289       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.1195e-06     	|  20000          	|  0.04239        	|  0.003182       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.1151e-06     	|  20000          	|  0.042302       	|  0.0031754      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00046293     	|  86             	|  0.039812       	|  0.0029885      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.2552e-06     	|  20086          	|  0.025212       	|  0.0018926      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.1675e-06     	|  20000          	|  0.02335        	|  0.0017528      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.1203e-06     	|  20086          	|  0.022503       	|  0.0016892      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.1085e-06     	|  20000          	|  0.022171       	|  0.0016642      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  9.4305e-07     	|  20000          	|  0.018861       	|  0.0014158      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  9.2256e-07     	|  20000          	|  0.018451       	|  0.001385       	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  8.5702e-07     	|  20000          	|  0.01714        	|  0.0012866      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.2993e-07     	|  20000          	|  0.016599       	|  0.001246       	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  8.1951e-07     	|  20000          	|  0.01639        	|  0.0012303      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  7.9989e-07     	|  20000          	|  0.015998       	|  0.0012009      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  7.5509e-07     	|  20000          	|  0.015102       	|  0.0011336      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0065558      	|  2              	|  0.013112       	|  0.00098423     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0073591      	|  1              	|  0.0073591      	|  0.00055241     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00028265     	|  20             	|  0.005653       	|  0.00042434     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00026462     	|  20             	|  0.0052925      	|  0.00039728     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  3.3573e-05     	|  86             	|  0.0028873      	|  0.00021674     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00081469     	|  2              	|  0.0016294      	|  0.00012231     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  8.0945e-05     	|  20             	|  0.0016189      	|  0.00012152     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00072805     	|  2              	|  0.0014561      	|  0.0001093      	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00057823     	|  2              	|  0.0011565      	|  8.6809e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.0010916      	|  1              	|  0.0010916      	|  8.1937e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00042064     	|  2              	|  0.00084127     	|  6.315e-05      	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.3098e-06     	|  86             	|  0.00062864     	|  4.7189e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00050299     	|  1              	|  0.00050299     	|  3.7757e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00045204     	|  1              	|  0.00045204     	|  3.3932e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.2059e-06     	|  86             	|  0.00036171     	|  2.7152e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.6067e-06     	|  86             	|  0.00031017     	|  2.3283e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.5158e-06     	|  86             	|  0.00030236     	|  2.2696e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00026133     	|  1              	|  0.00026133     	|  1.9617e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  0.00012982     	|  2              	|  0.00025965     	|  1.949e-05      	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  2.9414e-06     	|  86             	|  0.00025296     	|  1.8988e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.0528e-06     	|  86             	|  0.00017654     	|  1.3252e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.6808e-06     	|  86             	|  0.00014455     	|  1.0851e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.5736e-06     	|  20             	|  0.00013147     	|  9.8689e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.2939e-06     	|  86             	|  0.00011127     	|  8.3526e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.0664e-06     	|  20             	|  0.00010133     	|  7.6062e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  4.0464e-06     	|  22             	|  8.902e-05      	|  6.6823e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.3532e-06     	|  22             	|  7.3771e-05     	|  5.5376e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  3.2441e-06     	|  22             	|  7.1371e-05     	|  5.3575e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  3.4205e-05     	|  2              	|  6.8409e-05     	|  5.1351e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.0831e-06     	|  20             	|  6.1661e-05     	|  4.6286e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.5028e-06     	|  22             	|  5.5063e-05     	|  4.1333e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.3841e-06     	|  22             	|  5.245e-05      	|  3.9372e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.3739e-06     	|  22             	|  5.2225e-05     	|  3.9203e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.693e-06      	|  20             	|  3.3859e-05     	|  2.5416e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  3.1141e-05     	|  1              	|  3.1141e-05     	|  2.3376e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4065e-06     	|  22             	|  3.0943e-05     	|  2.3228e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.4032e-06     	|  22             	|  3.087e-05      	|  2.3172e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.311e-06      	|  20             	|  2.622e-05      	|  1.9682e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  9.0851e-06     	|  2              	|  1.817e-05      	|  1.3639e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  8.0345e-06     	|  2              	|  1.6069e-05     	|  1.2062e-06     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.451e-05      	|  1              	|  1.451e-05      	|  1.0892e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  7.1451e-06     	|  2              	|  1.429e-05      	|  1.0727e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  1.196e-05      	|  1              	|  1.196e-05      	|  8.9778e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.194e-05      	|  1              	|  1.194e-05      	|  8.9631e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  1.124e-05      	|  1              	|  1.124e-05      	|  8.4374e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  5.5102e-06     	|  2              	|  1.102e-05      	|  8.2724e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  9.4194e-06     	|  1              	|  9.4194e-06     	|  7.0707e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  8.5998e-06     	|  1              	|  8.5998e-06     	|  6.4555e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  7.2503e-06     	|  1              	|  7.2503e-06     	|  5.4425e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.6e-06        	|  2              	|  7.2001e-06     	|  5.4047e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  7.0408e-06     	|  1              	|  7.0408e-06     	|  5.2852e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  6.6999e-06     	|  1              	|  6.6999e-06     	|  5.0293e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  6.3898e-06     	|  1              	|  6.3898e-06     	|  4.7965e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.9248e-06     	|  2              	|  5.8496e-06     	|  4.391e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.8703e-06     	|  2              	|  5.7407e-06     	|  4.3092e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.8e-06        	|  2              	|  5.6e-06        	|  4.2037e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  5.2787e-06     	|  1              	|  5.2787e-06     	|  3.9625e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  5.1707e-06     	|  1              	|  5.1707e-06     	|  3.8814e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.0794e-06     	|  1              	|  5.0794e-06     	|  3.8129e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.8196e-06     	|  1              	|  4.8196e-06     	|  3.6178e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.8e-06        	|  1              	|  4.8e-06        	|  3.6031e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.5905e-06     	|  1              	|  4.5905e-06     	|  3.4459e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  4.4899e-06     	|  1              	|  4.4899e-06     	|  3.3703e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.2254e-06     	|  2              	|  4.4508e-06     	|  3.341e-07      	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  4.4405e-06     	|  1              	|  4.4405e-06     	|  3.3333e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  4.2599e-06     	|  1              	|  4.2599e-06     	|  3.1977e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.1197e-06     	|  2              	|  4.2394e-06     	|  3.1823e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.1053e-06     	|  2              	|  4.2105e-06     	|  3.1606e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.0494e-06     	|  1              	|  4.0494e-06     	|  3.0397e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  3.8296e-06     	|  1              	|  3.8296e-06     	|  2.8747e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  3.7905e-06     	|  1              	|  3.7905e-06     	|  2.8453e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  3.5493e-06     	|  1              	|  3.5493e-06     	|  2.6643e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.77e-06       	|  2              	|  3.54e-06       	|  2.6573e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.4096e-06     	|  1              	|  3.4096e-06     	|  2.5594e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  3.39e-06       	|  1              	|  3.39e-06       	|  2.5447e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.3192e-06     	|  1              	|  3.3192e-06     	|  2.4916e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.1497e-06     	|  1              	|  3.1497e-06     	|  2.3643e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  3.0398e-06     	|  1              	|  3.0398e-06     	|  2.2819e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  2.6803e-06     	|  1              	|  2.6803e-06     	|  2.012e-07      	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.2852e-06     	|  2              	|  2.5705e-06     	|  1.9295e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.4401e-06     	|  1              	|  2.4401e-06     	|  1.8316e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.4093e-06     	|  1              	|  2.4093e-06     	|  1.8086e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  2.3898e-06     	|  1              	|  2.3898e-06     	|  1.7939e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.3302e-06     	|  1              	|  2.3302e-06     	|  1.7491e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.3097e-06     	|  1              	|  2.3097e-06     	|  1.7338e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  1.6366e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  2.0303e-06     	|  1              	|  2.0303e-06     	|  1.524e-07      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.8999e-06     	|  1              	|  1.8999e-06     	|  1.4262e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.8394e-06     	|  1              	|  1.8394e-06     	|  1.3807e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.7993e-06     	|  1              	|  1.7993e-06     	|  1.3507e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.6298e-06     	|  1              	|  1.6298e-06     	|  1.2234e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
