The paper demonstrates that the Variational Online Gauss-Newton (VOGN) method of Khan et al. (2018) can be successfully scaled to deep learning architectures. The authors demonstrated the scalability of Bayesian methods to large scale data such as ImageNet. Extensive experiments on large scale data and models are provided. The main result is an adoption of an existing model (VOGN) to make it practical for deep learning.