FIT Profiler Report

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                          	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                           	|  -              	|  760947         	|  3145.2         	|  100 %          	|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                              	|  155.71         	|  20             	|  3114.3         	|  99.015         	|
|  run_training_batch                                                                                                                                              	|  0.15125        	|  20000          	|  3025.0         	|  96.176         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                      	|  0.15093        	|  20000          	|  3018.5         	|  95.972         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                    	|  0.035501       	|  20000          	|  710.02         	|  22.575         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                         	|  0.0104         	|  20000          	|  208.0          	|  6.6133         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                  	|  0.83687        	|  42             	|  35.149         	|  1.1175         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                       	|  0.0010243      	|  20000          	|  20.487         	|  0.65136        	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                    	|  0.00093582     	|  20000          	|  18.716         	|  0.59507        	|
|  [LightningModule]NeRFSystem.setup                                                                                                                               	|  6.0085         	|  1              	|  6.0085         	|  0.19103        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_epoch_end       	|  0.20371        	|  20             	|  4.0741         	|  0.12953        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                  	|  9.0468e-05     	|  20042          	|  1.8132         	|  0.057648       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                            	|  6.3301e-05     	|  20042          	|  1.2687         	|  0.040336       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                 	|  0.0090583      	|  42             	|  0.38045        	|  0.012096       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_batch_end       	|  1.894e-05      	|  20000          	|  0.3788         	|  0.012043       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                 	|  1.7853e-05     	|  20000          	|  0.35706        	|  0.011352       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                         	|  1.5503e-05     	|  20000          	|  0.31007        	|  0.0098584      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                 	|  0.051591       	|  2              	|  0.10318        	|  0.0032806      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                     	|  3.5703e-06     	|  20000          	|  0.071406       	|  0.0022703      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                      	|  3.1579e-06     	|  20000          	|  0.063157       	|  0.002008       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                              	|  2.2843e-06     	|  20000          	|  0.045686       	|  0.0014525      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                  	|  2.1639e-06     	|  20000          	|  0.043278       	|  0.001376       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                   	|  2.065e-06      	|  20000          	|  0.041299       	|  0.0013131      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                    	|  1.8794e-06     	|  20000          	|  0.037588       	|  0.0011951      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_after_backward        	|  1.7539e-06     	|  20000          	|  0.035078       	|  0.0011153      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                            	|  1.673e-06      	|  20042          	|  0.03353        	|  0.001066       	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                       	|  1.5698e-06     	|  20000          	|  0.031397       	|  0.00099823     	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                    	|  1.459e-06      	|  20000          	|  0.02918        	|  0.00092776     	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                      	|  1.3296e-06     	|  20000          	|  0.026591       	|  0.00084545     	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                	|  1.3049e-06     	|  20000          	|  0.026098       	|  0.00082976     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_batch_start     	|  1.2661e-06     	|  20000          	|  0.025323       	|  0.00080511     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_before_zero_grad      	|  1.2376e-06     	|  20000          	|  0.024752       	|  0.00078695     	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                     	|  1.231e-06      	|  20000          	|  0.024619       	|  0.00078275     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_before_optimizer_step 	|  1.2227e-06     	|  20000          	|  0.024454       	|  0.00077749     	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                   	|  1.2061e-06     	|  20000          	|  0.024121       	|  0.00076692     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                  	|  1.1578e-06     	|  20000          	|  0.023156       	|  0.00073621     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                   	|  1.1404e-06     	|  20000          	|  0.022808       	|  0.00072516     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_before_backward       	|  1.129e-06      	|  20000          	|  0.022579       	|  0.00071788     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                	|  1.0949e-06     	|  20000          	|  0.021898       	|  0.00069624     	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                             	|  9.9659e-07     	|  20042          	|  0.019974       	|  0.00063504     	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                 	|  9.467e-07      	|  20000          	|  0.018934       	|  0.00060199     	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                	|  9.2017e-07     	|  20000          	|  0.018403       	|  0.00058512     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                            	|  9.1506e-07     	|  20000          	|  0.018301       	|  0.00058187     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                  	|  8.5215e-07     	|  20000          	|  0.017043       	|  0.00054187     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                             	|  8.4831e-07     	|  20000          	|  0.016966       	|  0.00053942     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                   	|  0.00077662     	|  20             	|  0.015532       	|  0.00049384     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                               	|  0.0003535      	|  42             	|  0.014847       	|  0.00047205     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                	|  0.0039512      	|  2              	|  0.0079025      	|  0.00025125     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                    	|  0.00029471     	|  20             	|  0.0058943      	|  0.0001874      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                  	|  0.0002426      	|  20             	|  0.0048519      	|  0.00015426     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                	|  0.0042137      	|  1              	|  0.0042137      	|  0.00013397     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                             	|  5.9705e-05     	|  42             	|  0.0025076      	|  7.9727e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                            	|  0.00056679     	|  2              	|  0.0011336      	|  3.6041e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                     	|  0.00048361     	|  2              	|  0.00096722     	|  3.0752e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                           	|  0.00045681     	|  2              	|  0.00091361     	|  2.9047e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.setup                    	|  0.00065499     	|  1              	|  0.00065499     	|  2.0825e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                          	|  0.00061873     	|  1              	|  0.00061873     	|  1.9672e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                   	|  0.00023623     	|  2              	|  0.00047247     	|  1.5022e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                    	|  0.00029035     	|  1              	|  0.00029035     	|  9.2315e-06     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                        	|  0.00024253     	|  1              	|  0.00024253     	|  7.7111e-06     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                      	|  0.00022515     	|  1              	|  0.00022515     	|  7.1583e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                      	|  8.6977e-06     	|  20             	|  0.00017395     	|  5.5307e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                 	|  4.1298e-06     	|  42             	|  0.00017345     	|  5.5147e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                 	|  2.9951e-06     	|  42             	|  0.0001258      	|  3.9996e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                               	|  2.4912e-06     	|  42             	|  0.00010463     	|  3.3266e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                    	|  4.5464e-06     	|  20             	|  9.0928e-05     	|  2.891e-06      	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                              	|  1.9526e-06     	|  42             	|  8.2009e-05     	|  2.6074e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_batch_end  	|  1.5202e-06     	|  42             	|  6.385e-05      	|  2.03e-06       	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                           	|  1.5064e-06     	|  42             	|  6.3268e-05     	|  2.0115e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_end        	|  3.1182e-05     	|  2              	|  6.2363e-05     	|  1.9828e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                             	|  1.4496e-06     	|  42             	|  6.0883e-05     	|  1.9357e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_batch_start	|  1.3651e-06     	|  42             	|  5.7335e-05     	|  1.8229e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                  	|  2.0727e-06     	|  20             	|  4.1454e-05     	|  1.318e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_epoch_start     	|  1.8923e-06     	|  20             	|  3.7846e-05     	|  1.2033e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                	|  1.3274e-06     	|  20             	|  2.6547e-05     	|  8.4405e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                 	|  1.0967e-05     	|  1              	|  1.0967e-05     	|  3.4869e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                               	|  5.4641e-06     	|  2              	|  1.0928e-05     	|  3.4745e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                   	|  5.2201e-06     	|  2              	|  1.044e-05      	|  3.3193e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                 	|  9.2303e-06     	|  1              	|  9.2303e-06     	|  2.9347e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                             	|  4.2585e-06     	|  2              	|  8.5169e-06     	|  2.7079e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                     	|  3.7821e-06     	|  2              	|  7.5642e-06     	|  2.405e-07      	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                            	|  7.3081e-06     	|  1              	|  7.3081e-06     	|  2.3235e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                	|  3.5078e-06     	|  2              	|  7.0157e-06     	|  2.2306e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                            	|  6.9421e-06     	|  1              	|  6.9421e-06     	|  2.2072e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                 	|  3.0138e-06     	|  2              	|  6.0275e-06     	|  1.9164e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                       	|  2.9905e-06     	|  2              	|  5.981e-06      	|  1.9016e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_epoch_start	|  2.8764e-06     	|  2              	|  5.7528e-06     	|  1.829e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                              	|  2.8643e-06     	|  2              	|  5.7286e-06     	|  1.8213e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                               	|  2.8629e-06     	|  2              	|  5.7258e-06     	|  1.8205e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                        	|  5.6373e-06     	|  1              	|  5.6373e-06     	|  1.7923e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                             	|  5.2536e-06     	|  1              	|  5.2536e-06     	|  1.6703e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                           	|  2.244e-06      	|  2              	|  4.488e-06      	|  1.4269e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_end             	|  4.435e-06      	|  1              	|  4.435e-06      	|  1.4101e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                   	|  4.3903e-06     	|  1              	|  4.3903e-06     	|  1.3958e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                        	|  3.6815e-06     	|  1              	|  3.6815e-06     	|  1.1705e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                          	|  3.1535e-06     	|  1              	|  3.1535e-06     	|  1.0026e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_train_start           	|  2.9858e-06     	|  1              	|  2.9858e-06     	|  9.4931e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_start      	|  1.4347e-06     	|  2              	|  2.8694e-06     	|  9.123e-08      	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                          	|  2.8284e-06     	|  1              	|  2.8284e-06     	|  8.9927e-08     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                        	|  2.7306e-06     	|  1              	|  2.7306e-06     	|  8.6818e-08     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                              	|  2.685e-06      	|  1              	|  2.685e-06      	|  8.5367e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_validation_epoch_end  	|  1.2456e-06     	|  2              	|  2.4913e-06     	|  7.9208e-08     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                  	|  2.4522e-06     	|  1              	|  2.4522e-06     	|  7.7965e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                     	|  2.3907e-06     	|  1              	|  2.3907e-06     	|  7.601e-08      	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                             	|  1.125e-06      	|  2              	|  2.2501e-06     	|  7.1539e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                            	|  1.8515e-06     	|  1              	|  1.8515e-06     	|  5.8866e-08     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                              	|  1.8412e-06     	|  1              	|  1.8412e-06     	|  5.854e-08      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_fit_end               	|  1.7481e-06     	|  1              	|  1.7481e-06     	|  5.5579e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                      	|  1.7285e-06     	|  1              	|  1.7285e-06     	|  5.4957e-08     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                          	|  1.6708e-06     	|  1              	|  1.6708e-06     	|  5.3121e-08     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                	|  1.655e-06      	|  1              	|  1.655e-06      	|  5.2618e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                   	|  1.5711e-06     	|  1              	|  1.5711e-06     	|  4.9953e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.teardown                 	|  1.5479e-06     	|  1              	|  1.5479e-06     	|  4.9213e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None}.on_fit_start             	|  1.3588e-06     	|  1              	|  1.3588e-06     	|  4.3202e-08     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                            	|  1.2973e-06     	|  1              	|  1.2973e-06     	|  4.1247e-08     	|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
