· 11 min read

Training Neural Networks: Activation Functions, Backpropagation, and TensorFlow Implementation

Explore how neural networks are trained with gradient descent, softmax, and backpropagation using TensorFlow. Understand activation functions and multiclass classification techniques.

Training Neural Networks: Activation Functions, Backpropagation, and TensorFlow Implementation

2.Neural network training

TensorFlow Implementation

Train a Neural Network in TensorFlow

Screenshot 2024-03-28 at 8.20.09 PM.png

Training Details

Screenshot 2024-03-28 at 8.27.16 PM.png

1. Create the model

Screenshot 2024-03-28 at 8.28.21 PM.png

2. Loss and cost functions

Screenshot 2024-03-28 at 8.32.11 PM.png

3. Gradient descent

Screenshot 2024-03-28 at 8.32.37 PM.png

Screenshot 2024-03-28 at 8.33.49 PM.png

Alternatives to the sigmoid activation

ReLU (Rectified Linear Unit)

Screenshot 2024-03-28 at 8.43.38 PM.png

Choosing activation functions

Screenshot 2024-03-28 at 8.46.43 PM.png

Hidden Layer

Screenshot 2024-03-28 at 8.47.09 PM.png

Screenshot 2024-03-28 at 8.48.38 PM.png

Why do we need activation functions?

Screenshot 2024-03-28 at 8.52.12 PM.png

Screenshot 2024-03-28 at 8.53.31 PM.png

Screenshot 2024-03-28 at 8.53.46 PM.png

Multiclass

target y can take on more than two possible values

Screenshot 2024-03-30 at 8.32.07 PM.png

Screenshot 2024-03-30 at 8.33.18 PM.png

Softmax

Screenshot 2024-03-30 at 8.36.45 PM.png

Cost

Screenshot 2024-03-30 at 8.40.03 PM.png

Neural network with softmax output

Screenshot 2024-03-30 at 8.45.23 PM.png

Screenshot 2024-03-30 at 8.45.59 PM.png

Improved implementation of softmax

Screenshot 2024-03-30 at 8.49.58 PM.png

Screenshot 2024-03-30 at 8.50.35 PM.png

Screenshot 2024-03-30 at 8.50.56 PM.png

Screenshot 2024-03-30 at 8.51.26 PM.png

Classification with multiple outputs

Screenshot 2024-03-30 at 8.53.01 PM.png

Screenshot 2024-03-30 at 8.53.15 PM.png

Advanced Optimization

Gradient Descent

Screenshot 2024-03-30 at 9.16.25 PM.png

Adam Algorithm Intuition

Screenshot 2024-03-30 at 9.16.48 PM.png

Screenshot 2024-03-30 at 9.28.37 PM.png

Additional Layer Types

Screenshot 2024-03-30 at 9.31.33 PM.png

Screenshot 2024-03-30 at 9.33.01 PM.png

Screenshot 2024-03-30 at 9.33.22 PM.png

Back propagation

What is a derivative?

Screenshot 2024-03-30 at 9.41.09 PM.png

Screenshot 2024-03-30 at 9.41.22 PM.png

Screenshot 2024-03-30 at 9.42.36 PM.png

Screenshot 2024-03-30 at 9.43.31 PM.png

Screenshot 2024-03-30 at 9.44.58 PM.png

Computation graph

Screenshot 2024-03-30 at 9.46.44 PM.png

Screenshot 2024-03-30 at 9.47.02 PM.png

Screenshot 2024-03-30 at 9.47.35 PM.png

Screenshot 2024-03-30 at 9.47.51 PM.png

Larger neural network example

Screenshot 2024-03-30 at 9.49.36 PM.png

Screenshot 2024-03-30 at 9.50.01 PM.png