随机矩阵理论(XII): 深度学习

本文是随机矩阵理论系列的第十二篇文章，讲述了随机矩阵在深度学习中的应用，探究深度学习为什么如此有效。

深度学习

随机矩阵理论

非凸优化中的鞍点问题

几个神经网络

• 超限学习机(Extreme Learning Machine)
• 回声状态机(Echo State Machine)

研究现状

• J. Silverstein, Asymptotics Applied to a Neural Network, Biol. Cybernetics, 1976.
• Y. L. Cun., I. I. Kanter and S. A. Solla (1991). Eigenvalues of covariance matrices: application to neural-network learning. Physical Review Letters, 66(18), 2396.
• A. Auffinger, G.B. Arous and Jiri Cerny (2013). Random matrices and complexity of spin glasses. Communications on Pure & Applied Mathematics, 66(2), 165-201.⭐️
• Y.N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli and Y. Bengio (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. International Conference on Neural Information Processing Systems (Vol.111, pp.2933-2941). MIT Press.
• A. Choromanska, M. Henaff, M. Mathieu, G.B. Arous and Y. Lecun (2015). The loss surfaces of multilayer networks. 18th International Conference on Artificial Intelligence and Statistics (AISTATS) 2015, San Diego, CA, USA.⭐️
• A. Choromanska, Y. Lecun and G.B. Arous (2015). Open problem : the landscape of the loss surfaces of multilayer networks. JMLR: Workshop and Conference Proceedings vol 40:1–5, 2015.
• L. Sagun, V.U. Guney, G.B. Arous and Y. Lecun (2015). Explorations on high dimensional landscapes. ICLR 2015.
• Naftali Tishby and Noga Zaslavsky, Deep Learning and the Information Bottleneck Principle, Information Theory Workshop(ITW), IEEE, 1(2015)⭐️
• D.J. Im, M. Tao and K. Branson (2016). An empirical analysis of deep network loss surfaces.
• R. Couillet, G. Wainrib, H. Sevi and H.T. Ali (2016). The asymptotic performance of linear echo state neural networks. Journal of Machine Learning Research, 17(1), 6171-6205.
• R. Couillet, G. Wainrib, H.T. Ali and H. Sevi (2016). A random matrix approach to echo-state neural networks. International Conference on Machine Learning (pp.517-525). JMLR.org.⭐️
• C. Louart, Z. Liao and R. Couillet (2017). A random matrix approach to neural networks. The 33rd International Conference on Machine Learning, New York, NY, USA, 2016.⭐️
• M. Suzen, C. Weber and J.J. Cerda (2017). Spectral ergodicity in deep learning architectures via surrogate random matrices.⭐️⭐️
• Jeffrey Pennington and Yasaman Bahri, Geometry of Neural Network Loss Surfaces via Random Matrix Theory, ICML2017.⭐️⭐️
• Jeffrey Pennington and Pratik Worah, Nonlinear random matrix theory for deep learning, NIPS 2017.⭐️⭐️
• Jeffrey Pennington, Samuel S. Schoenholz and Surya Ganguli, Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, NIPS 2017. ⭐️⭐️
• Rene Vidal et al., Mathematics of deep learning.

Written on June 20th, 2017 by 李军