How does gradient descent work?
The update rule is one line: θ ← θ - η ∇L(θ). Here θ is the vector of model weights, L is the training loss, ∇L(θ) is its gradient (the partial derivative of the loss with respect to every weight), and η is the learning rate, the step size. In deep learning, backpropagation computes that gradient efficiently layer by layer, and the optimizer applies the step. Repeat for thousands to millions of steps. The gradient points uphill, so training subtracts it. The method only promises a local minimum, and training stops when held-out loss stops improving, the early-stopping guard against overfitting.
What are batch, stochastic and mini-batch gradient descent?
The variants differ in how much data estimates each gradient.
| Variant | Data per step | Trade-off |
|---|---|---|
| Batch | The whole training set | Exact gradient, slow and memory-bound on large data |
| Stochastic (SGD) | One example | Cheap steps, noisy path |
| Mini-batch | A small batch, often tens to thousands of examples | Fits GPU parallelism; the deep learning default |
Modern optimizers build on mini-batch descent. Adam keeps running averages of past gradients and their squares to adapt a step size per parameter (Kingma and Ba, “Adam”, revised 30 January 2017). In practice “gradient descent” means mini-batch steps with an adaptive optimizer, not the textbook full-batch loop.
Does ChatGPT use gradient descent?
In training, yes. The GPT-3 paper states that all versions were trained with Adam, using β1 = 0.9 and β2 = 0.95 (Brown et al., 2020, read 2 November 2026). OpenAI’s instruction-following models were then tuned with reinforcement learning from human feedback (Ouyang et al., 2022, read 2 November 2026), which still updates weights by gradient steps, this time on a reward signal; that stage is part of post-training. At inference nothing is trained. A conversation never moves the weights; the model adapts to the prompt through context alone, which is why changing its behavior durably needs fine-tuning or a new model version.
Is a learning rate of 0.001 good?
It is a sensible starting point, not a rule. PyTorch’s Adam ships with lr=0.001 as its default (PyTorch torch.optim.Adam, read 2 November 2026). Plain SGD often needs a larger rate, and fine-tuning a large pretrained model usually a smaller one, because big steps overwrite what the model already knows. Read the loss curve: a loss that spikes or diverges means the rate is too high, a slow crawl means too low. Tune the rate against validation loss, then confirm on a held-out eval dataset, because a lower training loss is not proof of a better model.