- In a nutshell: the deeper the network becomes, the harder the optimization problem becomes.
- To provably solve optimization problems for general neural networks with two or more layers, the algorithms that would be necessary hit some of the biggest open problems in computer science.
- In the post, I explore the “hardness” in optimizing neural networks and see what the theory has to say.
- The simplest neural network is the single-node perceptron , whose optimization problem is convex .
- The reasons for the success of deep learning go far beyond overcoming the optimization problem.

It’s easy to optimize simple neural networks, let’s say single layer perceptron. But, as network becomes deeper, the optmization problem becomes crucial. This article discusses about such optimization problems with deep neural networks.

Continue reading “The hard thing about deep learning”