FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  888241         	|  4093.7         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  192.53         	|  20             	|  3850.5         	|  94.06          	|
|  run_training_batch                                                                                                                                                                                         	|  0.16196        	|  20000          	|  3239.2         	|  79.127         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.16155        	|  20000          	|  3231.0         	|  78.928         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.038988       	|  20000          	|  779.76         	|  19.048         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.87518        	|  486            	|  425.34         	|  10.39          	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  230.99         	|  1              	|  230.99         	|  5.6426         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0074883      	|  20000          	|  149.77         	|  3.6585         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0048602      	|  20000          	|  97.204         	|  2.3745         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00052678     	|  20000          	|  10.536         	|  0.25736        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00018068     	|  20486          	|  3.7013         	|  0.090416       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.17499        	|  20             	|  3.4997         	|  0.085491       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.005029       	|  486            	|  2.4441         	|  0.059704       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00011147     	|  20486          	|  2.2836         	|  0.055783       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.0197e-05     	|  20000          	|  0.60393        	|  0.014753       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.9685e-05     	|  20000          	|  0.3937         	|  0.0096173      	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.7738e-05     	|  20000          	|  0.35477        	|  0.0086663      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00043043     	|  486            	|  0.20919        	|  0.0051101      	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.069735       	|  2              	|  0.13947        	|  0.003407       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  6.4796e-06     	|  20000          	|  0.12959        	|  0.0031657      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  5.3509e-06     	|  20000          	|  0.10702        	|  0.0026142      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  5.0936e-06     	|  20000          	|  0.10187        	|  0.0024885      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.6423e-06     	|  20000          	|  0.092846       	|  0.002268       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  4.4943e-06     	|  20000          	|  0.089885       	|  0.0021957      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  4.3844e-06     	|  20000          	|  0.087689       	|  0.0021421      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.2818e-06     	|  20000          	|  0.085635       	|  0.0020919      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  4.244e-06      	|  20000          	|  0.084881       	|  0.0020735      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  4.051e-06      	|  20000          	|  0.081019       	|  0.0019791      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  4.0302e-06     	|  20000          	|  0.080605       	|  0.001969       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  4.0153e-06     	|  20000          	|  0.080306       	|  0.0019617      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  3.9804e-06     	|  20000          	|  0.079608       	|  0.0019447      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  3.9712e-06     	|  20000          	|  0.079425       	|  0.0019402      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.9643e-06     	|  20000          	|  0.079286       	|  0.0019368      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  3.9384e-06     	|  20000          	|  0.078769       	|  0.0019242      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  3.9372e-06     	|  20000          	|  0.078744       	|  0.0019236      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  3.9118e-06     	|  20000          	|  0.078236       	|  0.0019112      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  3.9063e-06     	|  20000          	|  0.078127       	|  0.0019085      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  3.8874e-06     	|  20000          	|  0.077748       	|  0.0018992      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  3.8705e-06     	|  20000          	|  0.077409       	|  0.001891       	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  3.8623e-06     	|  20000          	|  0.077246       	|  0.001887       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  3.8591e-06     	|  20000          	|  0.077181       	|  0.0018854      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  3.5815e-06     	|  20486          	|  0.073371       	|  0.0017923      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  2.7271e-06     	|  20000          	|  0.054542       	|  0.0013323      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.6435e-06     	|  20000          	|  0.05287        	|  0.0012915      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.5601e-06     	|  20486          	|  0.052445       	|  0.0012811      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  2.377e-06      	|  20000          	|  0.047541       	|  0.0011613      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  2.3612e-06     	|  20000          	|  0.047223       	|  0.0011536      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  2.3016e-06     	|  20000          	|  0.046033       	|  0.0011245      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  2.2617e-06     	|  20000          	|  0.045233       	|  0.001105       	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  2.257e-06      	|  20000          	|  0.045139       	|  0.0011027      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  2.228e-06      	|  20000          	|  0.04456        	|  0.0010885      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  2.1341e-06     	|  20000          	|  0.042681       	|  0.0010426      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0043315      	|  2              	|  0.008663       	|  0.00021162     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  1.2762e-05     	|  486            	|  0.0062024      	|  0.00015151     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00024586     	|  20             	|  0.0049172      	|  0.00012012     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.0002443      	|  20             	|  0.0048859      	|  0.00011935     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.5929e-06     	|  486            	|  0.0036902      	|  9.0143e-05     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0029143      	|  1              	|  0.0029143      	|  7.119e-05      	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.8798e-06     	|  486            	|  0.0023716      	|  5.7933e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  4.6915e-06     	|  486            	|  0.00228        	|  5.5697e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  4.6138e-06     	|  486            	|  0.0022423      	|  5.4775e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00010646     	|  20             	|  0.0021292      	|  5.2012e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  4.3381e-06     	|  486            	|  0.0021083      	|  5.1503e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  3.9175e-06     	|  486            	|  0.0019039      	|  4.6509e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  3.008e-06      	|  486            	|  0.0014619      	|  3.5711e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00065667     	|  2              	|  0.0013133      	|  3.2082e-05     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  2.5844e-06     	|  486            	|  0.001256       	|  3.0682e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.0006227      	|  2              	|  0.0012454      	|  3.0422e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.0010058      	|  1              	|  0.0010058      	|  2.4569e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00048232     	|  2              	|  0.00096463     	|  2.3564e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00033562     	|  2              	|  0.00067123     	|  1.6397e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00048168     	|  1              	|  0.00048168     	|  1.1767e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  0.00047114     	|  1              	|  0.00047114     	|  1.1509e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00030087     	|  1              	|  0.00030087     	|  7.3497e-06     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00029424     	|  1              	|  0.00029424     	|  7.1876e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  7.5567e-06     	|  20             	|  0.00015113     	|  3.6919e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.9225e-06     	|  20             	|  0.00011845     	|  2.8935e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  4.8286e-06     	|  22             	|  0.00010623     	|  2.5949e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  4.3904e-06     	|  22             	|  9.6588e-05     	|  2.3595e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  3.9397e-06     	|  22             	|  8.6673e-05     	|  2.1172e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  3.8794e-06     	|  22             	|  8.5347e-05     	|  2.0849e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  3.8793e-06     	|  22             	|  8.5344e-05     	|  2.0848e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  3.8288e-06     	|  22             	|  8.4233e-05     	|  2.0576e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.1869e-05     	|  2              	|  8.3739e-05     	|  2.0456e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  4.009e-06      	|  20             	|  8.0179e-05     	|  1.9586e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  3.1429e-06     	|  22             	|  6.9143e-05     	|  1.689e-06      	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  2.8214e-06     	|  20             	|  5.6429e-05     	|  1.3784e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  2.4919e-06     	|  22             	|  5.4822e-05     	|  1.3392e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  2.228e-06      	|  20             	|  4.4561e-05     	|  1.0885e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  3.8761e-05     	|  1              	|  3.8761e-05     	|  9.4685e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.5609e-05     	|  2              	|  3.1219e-05     	|  7.6261e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.0651e-05     	|  2              	|  2.1302e-05     	|  5.2037e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.5154e-05     	|  1              	|  1.5154e-05     	|  3.7019e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  7.0883e-06     	|  2              	|  1.4177e-05     	|  3.4631e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  1.2361e-05     	|  1              	|  1.2361e-05     	|  3.0197e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  1.1804e-05     	|  1              	|  1.1804e-05     	|  2.8834e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.1315e-05     	|  1              	|  1.1315e-05     	|  2.7639e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.0896e-05     	|  1              	|  1.0896e-05     	|  2.6616e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  5.4124e-06     	|  2              	|  1.0825e-05     	|  2.6443e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  9.9875e-06     	|  1              	|  9.9875e-06     	|  2.4397e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  4.9593e-06     	|  2              	|  9.9186e-06     	|  2.4229e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  9.0096e-06     	|  1              	|  9.0096e-06     	|  2.2009e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  8.9388e-06     	|  1              	|  8.9388e-06     	|  2.1836e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  8.4508e-06     	|  1              	|  8.4508e-06     	|  2.0644e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  4.2254e-06     	|  2              	|  8.4508e-06     	|  2.0644e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  4.0517e-06     	|  2              	|  8.1034e-06     	|  1.9795e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  4.0159e-06     	|  2              	|  8.0317e-06     	|  1.962e-07      	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  8.0308e-06     	|  1              	|  8.0308e-06     	|  1.9618e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  3.8412e-06     	|  2              	|  7.6825e-06     	|  1.8767e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  7.6126e-06     	|  1              	|  7.6126e-06     	|  1.8596e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  7.6126e-06     	|  1              	|  7.6126e-06     	|  1.8596e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  3.8058e-06     	|  2              	|  7.6117e-06     	|  1.8594e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.5618e-06     	|  2              	|  7.1237e-06     	|  1.7402e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  7.1228e-06     	|  1              	|  7.1228e-06     	|  1.7399e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  6.6357e-06     	|  1              	|  6.6357e-06     	|  1.621e-07      	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  6.1458e-06     	|  1              	|  6.1458e-06     	|  1.5013e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  6.1449e-06     	|  1              	|  6.1449e-06     	|  1.5011e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  5.7267e-06     	|  1              	|  5.7267e-06     	|  1.3989e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  5.7258e-06     	|  1              	|  5.7258e-06     	|  1.3987e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.3819e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  2.8284e-06     	|  2              	|  5.6569e-06     	|  1.3819e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  5.6569e-06     	|  1              	|  5.6569e-06     	|  1.3819e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  4.7488e-06     	|  1              	|  4.7488e-06     	|  1.16e-07       	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  4.679e-06      	|  1              	|  4.679e-06      	|  1.143e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  2.3395e-06     	|  2              	|  4.679e-06      	|  1.143e-07      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  4.2608e-06     	|  1              	|  4.2608e-06     	|  1.0408e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  4.2599e-06     	|  1              	|  4.2599e-06     	|  1.0406e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  4.191e-06      	|  1              	|  4.191e-06      	|  1.0238e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  3.9116e-06     	|  1              	|  3.9116e-06     	|  9.5551e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  9.3845e-08     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  3.8417e-06     	|  1              	|  3.8417e-06     	|  9.3845e-08     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  3.8408e-06     	|  1              	|  3.8408e-06     	|  9.3822e-08     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.8859e-06     	|  2              	|  3.7719e-06     	|  9.2139e-08     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  3.7719e-06     	|  1              	|  3.7719e-06     	|  9.2139e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  3.7709e-06     	|  1              	|  3.7709e-06     	|  9.2116e-08     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  3.702e-06      	|  1              	|  3.702e-06      	|  9.0433e-08     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  3.282e-06      	|  1              	|  3.282e-06      	|  8.0172e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.9548e-06     	|  1              	|  1.9548e-06     	|  4.7753e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
