Contents

- 1 Deep Learning History Timeline
- 1.1 McCulloch Pitts Neuron – Beginning
- 1.2 Frank Rosenblatt creates Perceptron
- 1.3 The first Backpropagation Model
- 1.4 Backpropagation with Chain Rule
- 1.5 Birth of Multilayer Neural Network
- 1.6 The Fall of Perceptron
- 1.7 Backpropagation is computer coded
- 1.8 Neural Network goes Deep
- 1.9 Neocognitron – First CNN Architecture
- 1.10 Hopfield Network – Early RNN
- 1.11 Proposal for Backpropagation in ANN
- 1.12 Boltzmann Machine
- 1.13 NetTalk – ANN Learns Speech
- 1.14 Implementation of Backpropagation
- 1.15 Restricted Boltzmann Machine
- 1.16 CNN using Backpropagation
- 1.17 Universal Approximators Theorem
- 1.18 Vanishing Gradient Problem Appears
- 1.19 The Milestone of LSTM
- 1.20 Deep Belief Network
- 1.21 GPU Revolution Begins
- 1.22 ImageNet is launched
- 1.23 Combat for vanishing gradient
- 1.24 AlexNet Starts Deep Learning Boom
- 1.25 The birth of GANs
- 1.26 AlphaGo beats human
- 1.27 Trio win Turing Award

## Introduction

The world right now is seeing a global AI revolution across all industry. And one of the driving factor of this AI revolution is Deep Learning. Thanks to giants like Google and Facebook, Deep Learning now has become a popular term and people might think that it is a recent discovery. But you might be surprise to know that history of deep learning dates back to 1940s.

Indeed, deep learning has not appeared overnight, rather it has evolved slowly and gradually over seven decades. And behind this evolution, there are many machine learning researchers who worked with great determination even when no one believed that neural networks have any future.

This is our humble attempt to take you through the history of deep learning to relive the key discoveries made by the researchers and how all these small baby steps contributed to the modern era of deep learning boom.

# Deep Learning History Timeline

*Disclaimer-*

*There would be countless researchers whose results, directly or indirectly, would have contributed to the emergence and boom of deep learning. This article only attempts to discover a brief history of deep learning by highlighting some key moments and events. E**fforts have been made to reproduce the chronological events of deep learning history as accurately as possible. If you have any concerns or feedback, then please do write to us.*

*Sources-*

- https://news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon
- https://en.wikipedia.org/wiki/Frank_Rosenblatthttps://en.wikipedia.org/wiki/Perceptron
- http://alchessmist.blogspot.com/2009/06/stuart-dreyfus-on-mathematics-chess.html
- https://www.sciencedirect.com/science/article/pii/0022247X62900045?via%3Dihub
- https://en.wikipedia.org/wiki/Backpropagation
- https://www.gwern.net/docs/statistics/decision/1960-kelley.pdf
- https://en.wikipedia.org/wiki/AI_winter
- http://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_part1.html
- https://mailman.srv.cs.cmu.edu/pipermail/connectionists/2014-July/027158.html
- https://en.wikipedia.org/wiki/Alexey_Ivakhnenko
- https://www.abebooks.com/Perceptrons-Introduction-Computational-Geometry-Marvin-Minsky/30050854532/bd
- http://people.idsia.ch/~juergen/linnainmaa1970thesis.pdf
- http://personalpage.flsi.or.jp/fukushima/index-e.html
- https://en.wikipedia.org/wiki/Convolutional_neural_network#History
- https://bulletin.swarthmore.edu/bulletin-issue-archive/index.html%3Fp=336.html
- https://en.wikipedia.org/wiki/Hopfield_network
- http://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf
- http://www.andreykurenkov.com/writing/ai/a-brief-history-of-neural-nets-and-deep-learning/
- http://www.cs.toronto.edu/~hinton/absps/cogscibm.pdf
- http://www.scholarpedia.org/article/Boltzmann_machine
- https://medium.com/@tanaykarmarkar/explainable-restricted-boltzmann-machine-for-collaborative-filtering-6f011035352d
- https://link.springer.com/article/10.1007%2FBF02551274
- https://en.wikipedia.org/wiki/Universal_approximation_theorem#
- https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber
- https://en.wikipedia.org/wiki/Sepp_Hochreiter
- http://people.idsia.ch/~juergen/
- https://slideslive.com/38906590/deep-learning-is-revolutionizing-artificial-intelligence
- http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf
- https://en.wikipedia.org/wiki/Deep_belief_network
- https://www.quora.com/What-does-Andrew-Ng-think-about-Deep-Learning
- https://qz.com/1307091/the-inside-story-of-how-ai-got-good-enough-to-dominate-silicon-valley/
- https://en.wikipedia.org/wiki/AlexNethttps://papers.nips.cc/paper/5423-generative-adversarial-nets.pdfhttp://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf