1. Neural Networks

Neurons and the brain

Neural networks

Origins: Algorithms that try to mimic the brain, Used in the 1980’s and early 1990’s

Fell out of favor in the late 1990’s

Resurgence from around 2005

speech → images → text (NLP) → …

Demand Prediction

• example
  • one sigmoid function is an activation → a neuron

Each neuron is an activation

Neural network model

• Layer 1

• Layer 2

• Output

More complex neural network

• Layer 3

$a_j^{[l]}=g(\overrightarrow{w}_j^{[l]}\cdot\overrightarrow{a}^{[l-1]}+b_j^{[l]})$

• $a_j^{[l]}$: Activation value of layer $l$, unit(neuron)
• $G$ function: sigmoid, activation function
• $\overrightarrow{a}^{[l-1]}$: output of layer $l-1$ (previous layer)
• $\overrightarrow{w}_j^{[l]},b_j^{[l]}$: parameters w & b of layer $l$, unit $j$

Forward propagation

TensorFlow

Data in TensorFlow

NumPy vs TensorFlow

Building a neural network

Digit classification model

Neural network implementation in Python

General implementation of forward propagation

Is there a path to AGI?

• ANI - artificial narrow intelligence
  • smart speaker, self-driving car, web search…
• AGI - artificial general intelligence
  • do anything a human can do

How neural networks are implemented efficiently

Matrix multiplication

Dot products

Vector matrix multiplication

Matrix matrix multiplication

Matrix multiplication rules

Matrix multiplication code

Dense layer vectorized