What Is Batch Size in Machine Learning? How to Choose It

LLM foundationsModels and inferencePublished By Simon Budziak

Batch size is the number of training examples a model processes before it updates its weights once. Small batches give noisy but frequent updates and fit in little memory, while large batches give steadier updates and use hardware better, at the cost of memory and sometimes generalization.

How do batch size, iterations and epochs relate?

Training loops over the data many times. One pass over every example is an epoch. Inside an epoch the data is cut into batches, and each batch produces one weight update, called an iteration or step. Iterations per epoch equal the dataset size divided by the batch size.

Dataset Batch size Updates per epoch
10,000 examples 10,000 (full batch) 1
10,000 examples 100 (mini-batch) 100
10,000 examples 1 (stochastic) 10,000

This is the knob behind the three flavors of gradient descent: batch, mini-batch and stochastic. Most practical training is mini-batch.

What happens with a small versus a large batch?

Small batch Large batch
Gradient estimate Noisy Smooth
Updates per epoch Many Few
Memory needed Low High
Hardware use Poor on a GPU Good, through parallelism
Risk Unstable loss curve Settles in sharper minima that can generalize less well

Small batches add noise that can help. The noise from small batches acts as a mild regularizer, which can reduce overfitting. A large batch finishes an epoch faster on a GPU but often needs a higher learning rate and a warm-up period to train as well.

How do you choose a batch size?

Start at 32 or 64, then raise it to the largest value your memory allows and watch validation loss, not just training loss. Change the learning rate when you change the batch size, because the right step size depends on how noisy each update is. If memory is the limit, gradient accumulation runs several small batches and applies one update, giving a large effective batch. Watch validation loss, not just training loss. Pair the choice with early stopping so extra epochs do not memorize the training data.

Is batch size the same thing at inference time?

Batch size means something different at inference. There, batching groups several requests so the hardware serves them together, which raises throughput but adds waiting time per request. That is a serving trade-off between cost and latency, separate from training. See inference for how serving differs from training.

This entry was drafted with AI assistance.

Frequently asked questions

What is a good batch size to start with?

32 is a common starting point, with 64, 128 and 256 as the next values to try. Use the largest size that fits in memory without hurting validation results, and re-tune the learning rate when you change it.

What is the difference between batch size and epoch?

Batch size is how many examples go through the model per weight update. An epoch is one full pass over the whole training set, which takes dataset size divided by batch size updates.

Summarize this page with

Train your team to build this