AI summary by TwinMind
The First Neural Networks
Asianometry
18m 52s
Origins of Neural Networks (1943-1949)
- Warren McCulloch (neuroscientist/philosopher) & Walter Pitts (math prodigy) published 1943 paper on neuron networks
- McCulloch searched for "least psychic event" - settled on neuron's binary firing behavior
- Inspired by Turing's 1936 paper on logical machines
- Argued neural networks could perform logical operations, arithmetic, data management
- McCulloch-Pitts neuron model: weighted sum of inputs, threshold activation, binary output (1 or 0)
- Very simple compared to biological neurons
- Lacked transmission delays, synchronized firing, frequency modulation
- Donald Hebb (Canadian psychologist) proposed learning theory in 1949 book "The Organization of Behavior"
- "Cells that fire together wire together" - repeated co-firing strengthens connections
- Introduced concept of connection weights
- Connectionism movement emerged from this work
AI Field Emergence (1950s)
- Late 1940s/early 1950s: exciting computer developments
- EDVAC stored program computer (John Von Neumann consultation)
- 1955: John McCarthy (Dartmouth professor) proposed summer workshop on thinking machines
- November 1955: McCarthy, Marvin Minsky, Claude Shannon wrote proposal to Rockefeller Institute
- Coined term "artificial intelligence"
- Dartmouth Workshop (June-August 1956)
- Herbert Simon & Alan Newell presented Logic Theorist program
- Proved 38 theorems from Principia Mathematica
- Introduced Physical Symbol System theory
- Arthur Samuel (IBM) presented checkers program (1952)
- Programmed IBM 701 in machine assembly (pre-Fortran)
- 1955: added self-play capability with heuristic editing
- Coined term "machine learning"
- Alan Turing (1948) wrote about simulating entire neural networks
- Predated capable computers, received little discussion
The Perceptron Era (1957-1960)
- Frank Rosenblatt (psychologist, Minsky's high school classmate) created perceptron
- Cornell aeronautical laboratory, Office of Naval Research sponsorship
- Studied pilot landing psychology, applied probability theory to human memory/perception
- Perceptron Mark 1 (1957): statistical model in IBM 704 computer
- July 1958: premature press conference before hardware completion
- New York Times: "first electronic device to think as human brain"
- Could "perceive, recognize, identify surroundings without human training"
- Navy officers called it "human being without life"
- Mark 1 perceptron machine architecture:
- S units (sensory system): binary light/sound inputs
- A units (association system): single layer McCulloch-Pitts neurons with weighted sums
- R units (response system): motor output as lights for operator review
- Rosenblatt's training methodology:
- Random initial weights (like baby's brain)
- Show stimuli (e.g., letter X), get response
- Adjust weights based on correct/incorrect feedback
- Claimed finite training cycles for any learnable classification task
Perceptron Movement Growth (Late 1950s-1960s)
- Rosenblatt visited Stanford Research Institute (now SRI International)
- Bernard Widrow & student Ted Hoff created Adaline and Madaline devices
- Public impact: magazine articles asking if human brains would be replaced
- Rosenblatt's bold claims: perceptron "will be conscious of its own existence," "capable of having original ideas"
- 1960: Mark 1 machine completed and demonstrated
- 1962: Rosenblatt published "Principles of Neurodynamics" textbook
- Acknowledged failings of 1958 introduction
- Regretted impact of early rhetoric
- Clarified perceptron as class of models, not single thinking machine
Symbolic AI Opposition (1960s)
- Marvin Minsky & Seymour Papert became leading symbolic AI advocates
- Minsky argued "insurmountable limitations" of perceptron
- Motivations: personal slight over press, competition for research funding
- Mathematical criticisms circulated as drafts through second half of 1960s
- Core limitations identified:
- Single layer perceptrons only handle linearly separable problems
- XOR (exclusive or) problem: cannot solve with single linear separation
- Computational complexity scaling issues for complex images
- Need complete retraining for larger networks
- Symbolic algorithms could easily solve these problems
The Perceptron Book and AI Winter (1969-1971)
- 1969: Minsky & Papert published "Perceptrons" book
- Myth: book ended connectionism movement
- Reality: single layer limitations already known, multi-layer solutions existed
- Core problem: no training algorithm for multi-layer networks
- Rosenblatt's algorithm only worked for single layer
- Hidden layer training problem unsolved
- Rosenblatt experimented with frozen first layer, trained second layer
- Momentum stalled, funding shifted to symbolic AI approaches
- Stanford team abandoned connectionism
- Ted Hoff left Widrow's team for Intel, developed first microprocessor ideas
Rosenblatt's Later Work and Death (1968-1971)
- Shifted focus to chemical transfer of learned behaviors
- Experiments: trained flatworms in maze, ground up, fed chemicals to naive flatworms
- Involved in anti-war and anti-racist movements at Cornell
- Worked with student activists for university representation
- Democratic party political involvement
- 1971: died in tragic boating accident at age 43
- Three congressmen made tribute to his memory
- Minsky dedicated perceptron book to his memory
Legacy and Future Developments
- Government funding redirected from neural networks to symbolic AI
- Neural network research entered decade-long winter
- Later developments mentioned:
- Sigmoid output functions replaced binary outputs
- 1980s: backpropagation algorithm emerged for multi-layer training
- Rosenblatt coined "back propagating error correction" in 1962 (different from modern algorithm)
- Neural networks remained in shadows for decades until hardware/problems aligned
