How do batch size, iterations and epochs relate?
Training loops over the data many times. One pass over every example is an epoch. Inside an epoch the data is cut into batches, and each batch produces one weight update, called an iteration or step. Iterations per epoch equal the dataset size divided by the batch size.
| Dataset | Batch size | Updates per epoch |
|---|---|---|
| 10,000 examples | 10,000 (full batch) | 1 |
| 10,000 examples | 100 (mini-batch) | 100 |
| 10,000 examples | 1 (stochastic) | 10,000 |
This is the knob behind the three flavors of gradient descent: batch, mini-batch and stochastic. Most practical training is mini-batch.
What happens with a small versus a large batch?
| Small batch | Large batch | |
|---|---|---|
| Gradient estimate | Noisy | Smooth |
| Updates per epoch | Many | Few |
| Memory needed | Low | High |
| Hardware use | Poor on a GPU | Good, through parallelism |
| Risk | Unstable loss curve | Settles in sharper minima that can generalize less well |
Small batches add noise that can help. The noise from small batches acts as a mild regularizer, which can reduce overfitting. A large batch finishes an epoch faster on a GPU but often needs a higher learning rate and a warm-up period to train as well.
How do you choose a batch size?
Start at 32 or 64, then raise it to the largest value your memory allows and watch validation loss, not just training loss. Change the learning rate when you change the batch size, because the right step size depends on how noisy each update is. If memory is the limit, gradient accumulation runs several small batches and applies one update, giving a large effective batch. Watch validation loss, not just training loss. Pair the choice with early stopping so extra epochs do not memorize the training data.
Is batch size the same thing at inference time?
Batch size means something different at inference. There, batching groups several requests so the hardware serves them together, which raises throughput but adds waiting time per request. That is a serving trade-off between cost and latency, separate from training. See inference for how serving differs from training.