FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881073         	|  678.08         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  32.57          	|  20             	|  651.41         	|  96.066         	|
|  run_training_batch                                                                                                                                                                                         	|  0.019397       	|  20000          	|  387.94         	|  57.211         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.018934       	|  20000          	|  378.67         	|  55.844         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.0098261      	|  20000          	|  196.52         	|  28.982         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0096134      	|  20000          	|  192.27         	|  28.355         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0050066      	|  20000          	|  100.13         	|  14.767         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.88972        	|  38             	|  33.809         	|  4.986          	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00071104     	|  20000          	|  14.221         	|  2.0972         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  10.39          	|  1              	|  10.39          	|  1.5322         	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00022908     	|  20038          	|  4.5904         	|  0.67696        	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00015003     	|  20038          	|  3.0062         	|  0.44334        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.12445        	|  20             	|  2.4889         	|  0.36705        	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.024119       	|  38             	|  0.91654        	|  0.13517        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  4.2362e-05     	|  20000          	|  0.84725        	|  0.12495        	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.5134e-05     	|  20000          	|  0.50269        	|  0.074133       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.4465e-05     	|  20000          	|  0.4893         	|  0.072159       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.085629       	|  2              	|  0.17126        	|  0.025256       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  7.654e-06      	|  20000          	|  0.15308        	|  0.022575       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  4.9205e-06     	|  20000          	|  0.09841        	|  0.014513       	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.5026e-06     	|  20000          	|  0.090051       	|  0.01328        	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  4.3576e-06     	|  20000          	|  0.087153       	|  0.012853       	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  4.2563e-06     	|  20000          	|  0.085127       	|  0.012554       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  3.8383e-06     	|  20000          	|  0.076765       	|  0.011321       	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.6346e-06     	|  20000          	|  0.072693       	|  0.01072        	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.5871e-06     	|  20000          	|  0.071741       	|  0.01058        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  3.3904e-06     	|  20000          	|  0.067807       	|  0.0099998      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  3.3769e-06     	|  20000          	|  0.067538       	|  0.0099601      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.1838e-06     	|  20000          	|  0.063677       	|  0.0093907      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  3.0337e-06     	|  20000          	|  0.060674       	|  0.0089479      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.908e-06      	|  20000          	|  0.058159       	|  0.008577       	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.7734e-06     	|  20038          	|  0.055574       	|  0.0081958      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.7498e-06     	|  20000          	|  0.054996       	|  0.0081105      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.7363e-06     	|  20000          	|  0.054727       	|  0.0080708      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.7111e-06     	|  20000          	|  0.054221       	|  0.0079962      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.6827e-06     	|  20000          	|  0.053654       	|  0.0079126      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.6801e-06     	|  20000          	|  0.053602       	|  0.007905       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.6154e-06     	|  20000          	|  0.052308       	|  0.007714       	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.6026e-06     	|  20000          	|  0.052051       	|  0.0076762      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.586e-06      	|  20000          	|  0.05172        	|  0.0076274      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.4761e-06     	|  20000          	|  0.049521       	|  0.0073031      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  2.4038e-06     	|  20038          	|  0.048168       	|  0.0071036      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  2.0928e-06     	|  20000          	|  0.041856       	|  0.0061727      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.7058e-06     	|  20000          	|  0.034116       	|  0.0050313      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.612e-06      	|  20000          	|  0.03224        	|  0.0047546      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.5661e-06     	|  20000          	|  0.031322       	|  0.0046192      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.51e-06       	|  20000          	|  0.0302         	|  0.0044537      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.4609e-06     	|  20000          	|  0.029218       	|  0.0043089      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.4607e-06     	|  20000          	|  0.029213       	|  0.0043082      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.2654e-06     	|  20000          	|  0.025308       	|  0.0037323      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.224e-06      	|  20000          	|  0.024481       	|  0.0036103      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00049507     	|  38             	|  0.018813       	|  0.0027744      	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.014542       	|  1              	|  0.014542       	|  0.0021446      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0038901      	|  2              	|  0.0077802      	|  0.0011474      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00034394     	|  20             	|  0.0068787      	|  0.0010144      	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00029206     	|  20             	|  0.0058413      	|  0.00086144     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.0001302      	|  20             	|  0.0026041      	|  0.00038404     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.9544e-05     	|  38             	|  0.0018827      	|  0.00027764     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00055672     	|  2              	|  0.0011134      	|  0.0001642      	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00054427     	|  2              	|  0.0010885      	|  0.00016053     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00048933     	|  2              	|  0.00097865     	|  0.00014433     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00042375     	|  2              	|  0.00084749     	|  0.00012498     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00070345     	|  1              	|  0.00070345     	|  0.00010374     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00043142     	|  1              	|  0.00043142     	|  6.3623e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.0003223      	|  1              	|  0.0003223      	|  4.753e-05      	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  6.757e-06      	|  38             	|  0.00025677     	|  3.7866e-05     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00025674     	|  1              	|  0.00025674     	|  3.7862e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.627e-06      	|  38             	|  0.00013783     	|  2.0326e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.517e-06      	|  38             	|  0.00013364     	|  1.9709e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.6623e-06     	|  20             	|  0.00013325     	|  1.965e-05      	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.5736e-06     	|  20             	|  0.00011147     	|  1.6439e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  5.2354e-06     	|  20             	|  0.00010471     	|  1.5442e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.7202e-06     	|  38             	|  0.00010337     	|  1.5244e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.5477e-06     	|  38             	|  9.6813e-05     	|  1.4277e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.998e-06      	|  22             	|  8.7957e-05     	|  1.2971e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.2742e-06     	|  38             	|  8.6421e-05     	|  1.2745e-05     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.7729e-06     	|  22             	|  8.3004e-05     	|  1.2241e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  3.9849e-05     	|  2              	|  7.9699e-05     	|  1.1754e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  7.1479e-05     	|  1              	|  7.1479e-05     	|  1.0541e-05     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.6646e-06     	|  22             	|  5.8622e-05     	|  8.6453e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5356e-06     	|  38             	|  5.8354e-05     	|  8.6057e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.6168e-06     	|  22             	|  5.7571e-05     	|  8.4902e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.5295e-06     	|  22             	|  5.5649e-05     	|  8.2068e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.4636e-06     	|  22             	|  5.4198e-05     	|  7.9929e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.2803e-06     	|  38             	|  4.8651e-05     	|  7.1748e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.6601e-06     	|  22             	|  3.6522e-05     	|  5.386e-06      	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.6373e-06     	|  22             	|  3.6021e-05     	|  5.3121e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.7984e-06     	|  20             	|  3.5968e-05     	|  5.3043e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.6065e-06     	|  20             	|  3.213e-05      	|  4.7383e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  9.6802e-06     	|  2              	|  1.936e-05      	|  2.8552e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  6.8401e-06     	|  2              	|  1.368e-05      	|  2.0175e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.3079e-05     	|  1              	|  1.3079e-05     	|  1.9289e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.2361e-05     	|  1              	|  1.2361e-05     	|  1.8229e-06     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  5.5996e-06     	|  2              	|  1.1199e-05     	|  1.6516e-06     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  5.0049e-06     	|  2              	|  1.001e-05      	|  1.4762e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.0899e-06     	|  2              	|  8.1798e-06     	|  1.2063e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  7.7998e-06     	|  1              	|  7.7998e-06     	|  1.1503e-06     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  6.74e-06       	|  1              	|  6.74e-06       	|  9.9397e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  3.2503e-06     	|  2              	|  6.5006e-06     	|  9.5868e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.2298e-06     	|  2              	|  6.4597e-06     	|  9.5263e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  6.2604e-06     	|  1              	|  6.2604e-06     	|  9.2324e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  5.5004e-06     	|  1              	|  5.5004e-06     	|  8.1117e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.665e-06      	|  2              	|  5.33e-06       	|  7.8603e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.4796e-06     	|  2              	|  4.9593e-06     	|  7.3137e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.9407e-06     	|  1              	|  4.9407e-06     	|  7.2862e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  4.7805e-06     	|  1              	|  4.7805e-06     	|  7.05e-07       	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.3148e-06     	|  2              	|  4.6296e-06     	|  6.8275e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.5598e-06     	|  1              	|  4.5598e-06     	|  6.7245e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.5197e-06     	|  1              	|  4.5197e-06     	|  6.6654e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.2552e-06     	|  2              	|  4.5104e-06     	|  6.6517e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.4703e-06     	|  1              	|  4.4703e-06     	|  6.5926e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  2.2203e-06     	|  2              	|  4.4405e-06     	|  6.5487e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  4.3297e-06     	|  1              	|  4.3297e-06     	|  6.3852e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  4.0298e-06     	|  1              	|  4.0298e-06     	|  5.943e-07      	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  3.7402e-06     	|  1              	|  3.7402e-06     	|  5.5158e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  3.3099e-06     	|  1              	|  3.3099e-06     	|  4.8813e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  3.2699e-06     	|  1              	|  3.2699e-06     	|  4.8222e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  3.2196e-06     	|  1              	|  3.2196e-06     	|  4.7481e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.5353e-06     	|  2              	|  3.0706e-06     	|  4.5283e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  3.01e-06       	|  1              	|  3.01e-06       	|  4.439e-07      	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.9402e-06     	|  1              	|  2.9402e-06     	|  4.336e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.8601e-06     	|  1              	|  2.8601e-06     	|  4.2179e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.8387e-06     	|  1              	|  2.8387e-06     	|  4.1863e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.7902e-06     	|  1              	|  2.7902e-06     	|  4.1149e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.3653e-06     	|  2              	|  2.7306e-06     	|  4.027e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.6198e-06     	|  1              	|  2.6198e-06     	|  3.8635e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.5006e-06     	|  1              	|  2.5006e-06     	|  3.6877e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.4494e-06     	|  1              	|  2.4494e-06     	|  3.6122e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  2.4494e-06     	|  1              	|  2.4494e-06     	|  3.6122e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.1895e-06     	|  1              	|  2.1895e-06     	|  3.229e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.1793e-06     	|  1              	|  2.1793e-06     	|  3.2139e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.17e-06       	|  1              	|  2.17e-06       	|  3.2002e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.0796e-06     	|  1              	|  2.0796e-06     	|  3.0669e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.9399e-06     	|  1              	|  1.9399e-06     	|  2.8609e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9204e-06     	|  1              	|  1.9204e-06     	|  2.8321e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  1.9008e-06     	|  1              	|  1.9008e-06     	|  2.8032e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.7993e-06     	|  1              	|  1.7993e-06     	|  2.6535e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.6904e-06     	|  1              	|  1.6904e-06     	|  2.4928e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.5497e-06     	|  1              	|  1.5497e-06     	|  2.2854e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.4706e-06     	|  1              	|  1.4706e-06     	|  2.1687e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.4203e-06     	|  1              	|  1.4203e-06     	|  2.0945e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.37e-06       	|  1              	|  1.37e-06       	|  2.0204e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.18e-06       	|  1              	|  1.18e-06       	|  1.7402e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
