Don’t use deep learning your data isn’t that big · Simply Statistics

Don't use deep learning your data isn't that big

  • Don’t use deep learning your data isn’t that big

    The other day Brian was at a National Academies meeting and he gave one of his usual classic quotes:

    When I saw that quote I was reminded of the blog post Don’t use hadoop – your data isn’t that big.

  • Just as with Hadoop at the time that post was written – deep learning has achieved a level of mania and hype that means people are trying it for every problem.
  • The issue is that only a very few places actually have the data to do deep learning.
  • But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters.
  • If you are Google, Amazon, or Facebook and have near infinite data it makes sense to deep learn.

Best quote from NAS DS Round Table: “I mean, do we need deep learning to analyze 30 subjects?” – B Caffo @simplystats #datascienceinreallife
Continue reading “Don’t use deep learning your data isn’t that big · Simply Statistics”

The hard thing about deep learning

The hard thing about #DeepLearning is the #optimization problem

  • In a nutshell: the deeper the network becomes, the harder the optimization problem becomes.
  • To provably solve optimization problems for general neural networks with two or more layers, the algorithms that would be necessary hit some of the biggest open problems in computer science.
  • In the post, I explore the “hardness” in optimizing neural networks and see what the theory has to say.
  • The simplest neural network is the single-node perceptron , whose optimization problem is convex .
  • The reasons for the success of deep learning go far beyond overcoming the optimization problem.


It’s easy to optimize simple neural networks, let’s say single layer perceptron. But, as network becomes deeper, the optmization problem becomes crucial. This article discusses about such optimization problems with deep neural networks.

Continue reading “The hard thing about deep learning”

The hard thing about deep learning

The hard thing about deep learning  #ai

  • In a nutshell: the deeper the network becomes, the harder the optimization problem becomes.
  • The simplest neural network is the single-node perceptron , whose optimization problem is convex .
  • To provably solve optimization problems for general neural networks with two or more layers, the algorithms that would be necessary hit some of the biggest open problems in computer science.
  • There is a rich variety of optimization algorithms to handle convex optimization problems, and every few years a better polynomial-time algorithm for convex optimization is discovered.
  • Judd also shows that the problem remains NP-hard even if it only requires a network to produce the correct output for just two-thirds of the training examples, which implies that even approximately training a neural network is intrinsically difficult in the worst case.

Deeper neural nets often yield harder optimization problems.
Continue reading “The hard thing about deep learning”