Chapter 7: Neural Networks

🧠 Neural Networks are inspired by the human brain and form the core of Deep Learning.
They can learn complex patterns and power everything from image recognition to language models like ChatGPT.

---

🧠 What is a Neural Network?

A Neural Network is a series of connected layers of nodes (neurons) that transform input data to an output through mathematical operations.

---

🔹 1. Structure of a Neural Network

css
[Input Layer] ➝ [Hidden Layers] ➝ [Output Layer]

• Input Layer: Takes features (e.g., pixels, words)

• Hidden Layers: Do computations (deep = many layers)

• Output Layer: Gives prediction (class, value)

---

🔗 Each neuron performs:

$$y = \text{Activation}(W \cdot x + b)$$

• x = input

• W = weight

• b = bias

• Activation = nonlinear function

---

🔹 2. Types of Neural Networks

Type	Purpose	Example Use
Perceptron	Basic unit (1 neuron)	Logical operations
Multilayer Perceptron (MLP)	Fully connected layers	Tabular data, basic tasks
Convolutional NN (CNN)	Images	Face detection, X-rays
Recurrent NN (RNN)	Sequences	Text, speech, time-series

---

🔹 3. Activation Functions

Activation functions decide whether a neuron should fire or not (nonlinearity is essential!)

Function	Formula	Usage
Sigmoid	$\frac{1}{1+e^{-x}}$	Outputs between 0 and 1
ReLU	$\max(0, x)$	Fast, used in hidden layers
Tanh	$\frac{e^x - e^{-x}}{e^x + e^{-x}}$	Outputs between -1 and 1
Softmax	Converts scores to probabilities	Used in final classification layer

---

📉 4. Loss Functions

Measures how far the predicted output is from the actual value.

Loss Function	Used For	Formula (Concept)
MSE	Regression	Mean Squared Error
Cross-Entropy	Classification	Log loss of predicted probability

---

🔄 5. Backpropagation & Gradient Descent

The learning process of a neural network.

• Forward Pass: Compute prediction

• Loss Computation

• Backward Pass (Backpropagation): Calculate gradients

• Gradient Descent: Update weights to reduce error

Formula:

$$W = W - \alpha \cdot \frac{\partial L}{\partial W}$$

• $\alpha$: learning rate

• $\frac{\partial L}{\partial W}$: gradient of loss wrt weights

---

🧪 6. Training a Neural Network

Steps:

1. Initialize weights

2. Feed input (Forward Pass)

3. Calculate loss

4. Backpropagate to get gradients

5. Update weights

6. Repeat for many epochs (iterations)

---

📌 Neural Network Terminology

Term	Meaning
Epoch	One full pass over training data
Batch Size	Number of samples per update
Learning Rate	How fast weights are updated
Overfitting	Model fits training too well
Regularization	Prevent overfitting (e.g., dropout, L2)

---

💻 Example (Simple MLP with Keras):

python
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(16, input_shape=(10,), activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=20, batch_size=32)

---

💡 Where Neural Networks Are Used:

• Facial recognition

• Voice assistants (Siri, Alexa)

• ChatGPT / BERT / LLMs

• Medical diagnosis

• Fraud detection

---

🧠 Summary of Chapter 7

Concept	Description
Neural Networks	Simulate brain-like learning
Layers	Input ➝ Hidden ➝ Output
Activation	Adds nonlinearity
Loss & Gradient	Used for learning
Backpropagation	Optimizes weights
Tools	TensorFlow, Keras, PyTorch

---

✅ Mini Assignment

1. Build a neural network using Keras to classify MNIST digits.

2. Try different activation functions and observe training accuracy.

3. Implement a 2-layer MLP from scratch using NumPy.