Table of Contents
Gradient descent is an optimization algorithm used to minimize a function by iteratively moving towards the lowest point. It is widely used in machine learning to optimize models by adjusting parameters to reduce error. This article explains the step-by-step calculations involved in applying gradient descent for machine learning tasks.
Understanding the Gradient Descent Algorithm
The core idea of gradient descent is to update model parameters in the direction of the negative gradient of the loss function. This process continues until the parameters converge to a minimum point, ideally the global minimum.
Step-by-Step Calculation Process
Suppose we have a simple linear regression model with a loss function, such as Mean Squared Error (MSE). The steps for applying gradient descent are as follows:
- Initialize parameters (e.g., weights and bias) with small random values.
- Calculate the predicted output using current parameters.
- Compute the loss function value based on predictions and actual data.
- Calculate the gradient of the loss function with respect to each parameter.
- Update each parameter by subtracting the product of the learning rate and the corresponding gradient.
This process repeats for a set number of iterations or until the change in loss becomes negligible.
Example Calculation
Consider a single data point with input x = 2 and output y = 4. Initialize weight w = 0.5 and bias b = 0. Use a learning rate of 0.1.
Calculate prediction: ŷ = wx + b = 0.5 * 2 + 0 = 1
Compute error: error = ŷ – y = 1 – 4 = -3
Calculate gradients:
Gradient w.r.t. weight: ∂L/∂w = 2 * error * x = 2 * (-3) * 2 = -12
Gradient w.r.t. bias: ∂L/∂b = 2 * error = 2 * (-3) = -6
Update parameters:
New weight: w = 0.5 – 0.1 * (-12) = 0.5 + 1.2 = 1.7
New bias: b = 0 – 0.1 * (-6) = 0 + 0.6 = 0.6