What Are Model Parameters in AI? Weights, Biases, Settings

LLM foundationsModels and inferencePublished By Simon Budziak

Model parameters are the numbers an AI model learns during training, mainly weights and biases. They determine how input becomes output. A language model with billions of parameters stores what it learned in those numbers, and you do not set them by hand.

What are weights and biases?

In a neural network, each connection has a weight that scales a signal, and each unit has a bias that shifts it. Training starts with random values and adjusts every one to reduce error, using a loss function and gradient descent. After training the parameters are frozen, and they are the model. Sharing a model means sharing these numbers, which is what an open-weight release is.

A tiny worked example: a layer with 3 inputs and 2 outputs has 3 x 2 = 6 weights plus 2 biases, so 8 parameters. Count it for every layer and a network with millions or billions of them results.

How do parameters differ from hyperparameters and settings?

Kind Who sets it When Examples
Parameters Training, automatically During training Weights, biases
Hyperparameters You Before training Learning rate, batch size, number of layers
Request settings You Each call Temperature, maximum output length

People searching “model parameters” often mean the third row: the dials on an API call. Those change how a trained model samples its answer and do not change what it learned.

What does parameter count tell you?

Parameter count is a rough measure of capacity and cost. More parameters can store more patterns, and they take more memory and compute to run: a rule of thumb is parameters times bytes per number gives the memory for the weights alone, so a 7 billion parameter model at 2 bytes each needs about 14 GB before anything else. Count is a poor measure of quality. A smaller model trained well on good data can beat a bigger one, and mixture-of-experts models use only a share of their parameters for each token, so total and active counts differ.

How do you shrink a model’s parameters?

Quantization stores each parameter in fewer bits, cutting memory with a small quality loss. Model distillation trains a smaller model to imitate a larger one. A small language model starts with fewer parameters by design. Choose the smallest model that passes your own evaluation, because every extra parameter is paid for on every request.

This entry was drafted with AI assistance.

Frequently asked questions

How many parameters do LLMs have?

From under a billion for small language models to hundreds of billions or more for frontier models. Treat a count without a named source as an estimate.

What are parameter settings?

Usually they mean request settings such as temperature, which you choose per call. They are not learned parameters. Training hyperparameters are a third group, set before training.

Summarize this page with

Train your team to build this