Training Data in Machine Learning: Examples, Splits and LLMs

LLM foundationsRetrieval and dataPublished By Simon Budziak

Training data is the set of examples a machine learning model learns its parameters from. For a spam filter that means emails paired with spam or not spam labels; for a large language model it means trillions of tokens of text. A model can only learn patterns its training data contains, so coverage and quality cap what it can do.

What are training, validation and test data?

They are three disjoint slices of your data, each with one job.

Set Used for How often the team looks What goes wrong if misused
Training Fitting the model’s parameters Constantly Memorized noise, overfitting
Validation Tuning settings, choosing a model, early stopping Every experiment Settings tuned to one lucky slice
Test One final estimate of real performance Once per decision Scores that do not survive production

A test set used to make decisions stops being a test set. The same rule holds for LLM eval datasets: when public test questions leak into training data, scores inflate, the problem called benchmark contamination.

What is an example of training data?

One row of input plus whatever the model should learn to produce.

Task One training example Where the target comes from
Spam filter An email’s text and sender Users marking messages as spam
Invoice amount prediction Line items, supplier, date The amount actually paid
LLM pretraining A passage of web text The next token in the passage itself
Instruction fine-tuning A prompt A response written by a person

The first two are supervised learning with labels people or systems recorded. Pretraining is self-supervised: the text is its own answer key, which is why its scale is enormous. Meta’s Llama 3 “is pretrained on over 15T tokens that were all collected from publicly available sources”, a dataset “seven times larger than that used for Llama 2” (Meta AI blog, 18 April 2024, read 16 November 2026).

What makes training data good?

It looks like the data the model will meet in production, and its labels are right. Coverage beats volume: a million examples from one region teach nothing about another. Labels need a written guideline and review, the work of data annotation, and the labels you trust become your ground truth. Gaps can be filled with synthetic data, validated against real cases.

For high-risk systems the EU AI Act makes this a legal duty. Article 10(3) requires training, validation and testing data sets to be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete”, applying from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems (AI Act Explorer, Article 10, read 16 November 2026). Write down where every training set came from; you will be asked.

Do you need your own training data to use an LLM?

Rarely, because most LLM projects never train a model at all. Prompting and RAG use a pretrained model unchanged, and your documents reach it as context, not as training data. Your examples become training data only when you fine-tune. What every LLM project does need is evaluation data from its own domain. Model providers, meanwhile, owe transparency: Article 53(1)(d) requires general-purpose model providers to publish “a sufficiently detailed summary about the content used for training”, in force since 2 August 2025 (AI Act Explorer, Article 53, read 16 November 2026), using a Commission template published on 24 July 2025 (European Commission, read 16 November 2026).

This entry was drafted with AI assistance.

Frequently asked questions

What are the three main types of datasets in machine learning?

Training data, which the model learns from; validation data, used to tune settings and choose between models; and test data, held back for one final, honest score.

What is the difference between training data and test data?

Training data is what the model learns from. Test data is held back and used once to estimate how the model performs on examples it has never seen, so mixing the two inflates the score.

Summarize this page with

Train your team to build this