FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  885105         	|  1789.5         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  80.142         	|  20             	|  1602.8         	|  89.567         	|
|  run_training_batch                                                                                                                                                                                         	|  0.061118       	|  20000          	|  1222.4         	|  68.306         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.060764       	|  20000          	|  1215.3         	|  67.91          	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.017308       	|  20000          	|  346.16         	|  19.344         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.63659        	|  290            	|  184.61         	|  10.316         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  176.02         	|  1              	|  176.02         	|  9.8357         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.008117       	|  20000          	|  162.34         	|  9.0716         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0022576      	|  20000          	|  45.152         	|  2.5231         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00052405     	|  20000          	|  10.481         	|  0.58567        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00018212     	|  20290          	|  3.6952         	|  0.20649        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.15242        	|  20             	|  3.0483         	|  0.17034        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00010952     	|  20290          	|  2.2221         	|  0.12417        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.003222       	|  290            	|  0.93437        	|  0.052213       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.6399e-05     	|  20000          	|  0.52799        	|  0.029504       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.0731e-05     	|  20000          	|  0.41463        	|  0.023169       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.9827e-05     	|  20000          	|  0.39655        	|  0.022159       	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00047976     	|  290            	|  0.13913        	|  0.0077746      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.3791e-06     	|  20000          	|  0.12758        	|  0.0071293      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  5.8043e-06     	|  20000          	|  0.11609        	|  0.0064869      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.5469e-06     	|  20000          	|  0.11094        	|  0.0061992      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  5.2539e-06     	|  20000          	|  0.10508        	|  0.0058718      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  5.051e-06      	|  20000          	|  0.10102        	|  0.005645       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.7457e-06     	|  20000          	|  0.094915       	|  0.0053038      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  4.6684e-06     	|  20290          	|  0.094722       	|  0.0052931      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.6843e-06     	|  20000          	|  0.093687       	|  0.0052352      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.5647e-06     	|  20000          	|  0.091295       	|  0.0051016      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.3422e-06     	|  20000          	|  0.086844       	|  0.0048529      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  4.1437e-06     	|  20000          	|  0.082874       	|  0.004631       	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  4.1259e-06     	|  20000          	|  0.082517       	|  0.0046111      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.1e-06        	|  20000          	|  0.082001       	|  0.0045822      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  4.098e-06      	|  20000          	|  0.081961       	|  0.00458        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  4.0569e-06     	|  20000          	|  0.081138       	|  0.004534       	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  4.0512e-06     	|  20000          	|  0.081024       	|  0.0045276      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  4.0405e-06     	|  20000          	|  0.080811       	|  0.0045157      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  4.0403e-06     	|  20000          	|  0.080807       	|  0.0045155      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  4.0389e-06     	|  20000          	|  0.080777       	|  0.0045138      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  4.011e-06      	|  20000          	|  0.080219       	|  0.0044827      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  4.0058e-06     	|  20000          	|  0.080115       	|  0.0044768      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.977e-06      	|  20000          	|  0.079541       	|  0.0044447      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  3.9625e-06     	|  20000          	|  0.07925        	|  0.0044285      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.038423       	|  2              	|  0.076847       	|  0.0042942      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  3.1287e-06     	|  20290          	|  0.06348        	|  0.0035473      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  2.8566e-06     	|  20000          	|  0.057132       	|  0.0031925      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  2.8204e-06     	|  20000          	|  0.056409       	|  0.0031521      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  2.7706e-06     	|  20000          	|  0.055412       	|  0.0030964      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.693e-06      	|  20000          	|  0.053859       	|  0.0030097      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  2.6371e-06     	|  20000          	|  0.052742       	|  0.0029472      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  2.4965e-06     	|  20000          	|  0.04993        	|  0.0027901      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  2.4712e-06     	|  20000          	|  0.049424       	|  0.0027618      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  2.4071e-06     	|  20000          	|  0.048142       	|  0.0026902      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  2.4062e-06     	|  20000          	|  0.048123       	|  0.0026891      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0070051      	|  2              	|  0.01401        	|  0.00078289     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00057841     	|  20             	|  0.011568       	|  0.00064643     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0077354      	|  1              	|  0.0077354      	|  0.00043225     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00029294     	|  20             	|  0.0058588      	|  0.00032739     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00029288     	|  20             	|  0.0058575      	|  0.00032732     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  1.9335e-05     	|  290            	|  0.0056071      	|  0.00031332     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.9033e-06     	|  290            	|  0.002292       	|  0.00012807     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00093576     	|  2              	|  0.0018715      	|  0.00010458     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  5.3964e-06     	|  290            	|  0.001565       	|  8.745e-05      	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  5.0073e-06     	|  290            	|  0.0014521      	|  8.1145e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  4.939e-06      	|  290            	|  0.0014323      	|  8.0037e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  4.643e-06      	|  290            	|  0.0013465      	|  7.524e-05      	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00065126     	|  2              	|  0.0013025      	|  7.2785e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  3.9165e-06     	|  290            	|  0.0011358      	|  6.3468e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00053557     	|  2              	|  0.0010711      	|  5.9855e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  3.3583e-06     	|  290            	|  0.00097392     	|  5.4423e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00096903     	|  1              	|  0.00096903     	|  5.415e-05      	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00048354     	|  2              	|  0.00096708     	|  5.404e-05      	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  2.7702e-06     	|  290            	|  0.00080335     	|  4.4891e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00059511     	|  1              	|  0.00059511     	|  3.3255e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00043203     	|  1              	|  0.00043203     	|  2.4142e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00029193     	|  1              	|  0.00029193     	|  1.6313e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  7.7906e-06     	|  20             	|  0.00015581     	|  8.7067e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  6.516e-06      	|  20             	|  0.00013032     	|  7.2823e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  5.5046e-06     	|  22             	|  0.0001211      	|  6.7672e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.9808e-06     	|  22             	|  0.00010958     	|  6.1232e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.6584e-05     	|  2              	|  9.3168e-05     	|  5.2062e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  4.2125e-06     	|  22             	|  9.2675e-05     	|  5.1787e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  4.6129e-06     	|  20             	|  9.2259e-05     	|  5.1554e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  4.146e-06      	|  22             	|  9.1213e-05     	|  5.097e-06      	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  3.9999e-06     	|  22             	|  8.7998e-05     	|  4.9173e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  3.911e-06      	|  22             	|  8.6042e-05     	|  4.808e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  3.6252e-06     	|  22             	|  7.9755e-05     	|  4.4567e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  3.257e-06      	|  22             	|  7.1653e-05     	|  4.004e-06      	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  3.4327e-06     	|  20             	|  6.8653e-05     	|  3.8364e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  3.2896e-06     	|  20             	|  6.5792e-05     	|  3.6765e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  3.2196e-05     	|  1              	|  3.2196e-05     	|  1.7991e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  1.2257e-05     	|  2              	|  2.4514e-05     	|  1.3699e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  9.2192e-06     	|  2              	|  1.8438e-05     	|  1.0303e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  8.7647e-06     	|  2              	|  1.7529e-05     	|  9.7954e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  7.5782e-06     	|  2              	|  1.5156e-05     	|  8.4694e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  6.6701e-06     	|  2              	|  1.334e-05      	|  7.4545e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  5.6578e-06     	|  2              	|  1.1316e-05     	|  6.3231e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  1.1315e-05     	|  1              	|  1.1315e-05     	|  6.3226e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  9.9177e-06     	|  1              	|  9.9177e-06     	|  5.542e-07      	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  9.8478e-06     	|  1              	|  9.8478e-06     	|  5.503e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  4.9239e-06     	|  2              	|  9.8478e-06     	|  5.503e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  4.9239e-06     	|  2              	|  9.8478e-06     	|  5.503e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  9.4995e-06     	|  1              	|  9.4995e-06     	|  5.3083e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  9.3589e-06     	|  1              	|  9.3589e-06     	|  5.2297e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  9.0105e-06     	|  1              	|  9.0105e-06     	|  5.0351e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  9.0096e-06     	|  1              	|  9.0096e-06     	|  5.0346e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  4.2254e-06     	|  2              	|  8.4508e-06     	|  4.7223e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  4.0852e-06     	|  2              	|  8.1705e-06     	|  4.5657e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  4.0159e-06     	|  2              	|  8.0317e-06     	|  4.4881e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  8.0308e-06     	|  1              	|  8.0308e-06     	|  4.4876e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  7.5437e-06     	|  1              	|  7.5437e-06     	|  4.2154e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  7.5428e-06     	|  1              	|  7.5428e-06     	|  4.2149e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.7714e-06     	|  2              	|  7.5428e-06     	|  4.2149e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  7.5419e-06     	|  1              	|  7.5419e-06     	|  4.2144e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  7.0538e-06     	|  1              	|  7.0538e-06     	|  3.9417e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  6.6347e-06     	|  1              	|  6.6347e-06     	|  3.7075e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  6.2166e-06     	|  1              	|  6.2166e-06     	|  3.4738e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  6.1467e-06     	|  1              	|  6.1467e-06     	|  3.4348e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  3.0729e-06     	|  2              	|  6.1458e-06     	|  3.4343e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  6.0759e-06     	|  1              	|  6.0759e-06     	|  3.3952e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  3.038e-06      	|  2              	|  6.0759e-06     	|  3.3952e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  5.6578e-06     	|  1              	|  5.6578e-06     	|  3.1616e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  3.1611e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  5.2378e-06     	|  1              	|  5.2378e-06     	|  2.9269e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  5.1679e-06     	|  1              	|  5.1679e-06     	|  2.8878e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  4.3297e-06     	|  1              	|  4.3297e-06     	|  2.4194e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  4.2608e-06     	|  1              	|  4.2608e-06     	|  2.3809e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  4.2599e-06     	|  1              	|  4.2599e-06     	|  2.3804e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.19e-06       	|  1              	|  4.19e-06       	|  2.3414e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.19e-06       	|  1              	|  4.19e-06       	|  2.3414e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  3.9106e-06     	|  1              	|  3.9106e-06     	|  2.1853e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  2.1467e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  3.7728e-06     	|  1              	|  3.7728e-06     	|  2.1082e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  2.1077e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  2.1077e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  2.1077e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  2.1077e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  2.1072e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  2.1072e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  2.1072e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  2.1072e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  2.1072e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  3.702e-06      	|  1              	|  3.702e-06      	|  2.0687e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  3.3518e-06     	|  1              	|  3.3518e-06     	|  1.873e-07      	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  2.3041e-06     	|  1              	|  2.3041e-06     	|  1.2875e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
