The field of Artificial Intelligence (AI), which can definitely be considered to be the parent field of deep
learning, has a rich history going back to 1950. While we will not cover this history in much detail, we will go
over some of the key turning points in the field, which will lead us to deep learning.
Tasks that AI focused on in its early days were tasks that could be easily described formally, like the
game of checkers or chess. This notion of being able to easily describe the task formally is at the heart of what
can or cannot be done easily by a computer program. For instance, consider the game of chess. The formal
description of the game of chess would be the representation of the board, a description of how each of the
pieces move, the starting configuration, and a description of the configuration wherein the game terminates.
With these notions formalized, it's relatively easy to model a chess-playing AI program as a search and,
given sufficient computational resources, it’s possible to produces a relatively good chess-playing AI.
The first era of AI focused on such tasks with a fair amount of success. At the heart of the methodology
was a symbolic representation of the domain and the manipulation of symbols based on given rules (with
increasingly sophisticated algorithms for searching the solution space to arrive at a solution).
It must be noted that the formal definitions of such rules were done manually. However, such early
AI systems were fairly general purpose task/problem solvers in the sense that any problem that could be
described formally could be solved with the generic approach.
The key limitation about such systems is that the game of chess is a relatively easy problem for AI simply
because the problem setting is relatively simple and can be easily formalized. This is not the case with many of
the problems human beings solve on a day-to-day basis (natural intelligence). For instance, consider diagnosing
a disease (as a physician does) or transcribing human speech to text. These tasks, like most other tasks human
beings master easily, are hard to describe formally and presented a challenge in the early days of AI.
Human beings address such tasks by leveraging a large amount of knowledge about the task/problem
domain. Given this observation, subsequent AI systems relied on a large knowledge base which captured the
knowledge about the problem/task domain. One point to be noted is the term used here is knowledge, not
information or data. By knowledge we simply mean data/information that a program/algorithm can reason
about. An example of this could be a graph representation of a map with edges labeled with distances and
about traffic (which is being constantly updated), which allows a program to reason about the shortest path
between points.