What are the steps of data cleaning?
- Profile first: count nulls, distinct values and outliers per column before changing anything.
- Remove exact and near duplicates, deciding which record survives.
- Standardize formats: dates, currencies, units, country codes, casing.
- Handle missing values: fill only where a rule justifies it, otherwise flag.
- Validate against rules, such as a tax number checksum or a date range, and route failures to a person.
- Log every change so the cleaned set can be rebuilt from the raw one.
Cleaning is one stage of a data pipeline, and it should run there on every load, not once in a notebook.
How is data cleaning different for LLMs and RAG?
Tabular cleaning fixes values. Text cleaning for a RAG system fixes documents: it strips navigation and boilerplate, removes outdated versions so the assistant cannot quote last year’s policy, deduplicates near-identical files and keeps metadata such as owner and date for filtering. For retrieval, deleting a stale document is often worth more than any tuning of the model. What survives then goes to document chunking.
How do we deduplicate terms before measuring them?
Before we pay to measure a list of candidate glossary terms, we clean the list in two passes. The first pass is mechanical: each term is turned into a URL slug, then normalized by dropping a small set of stopwords such as “ai”, “the” and “in”, so “AI in finance” and “finance AI” collide, and the published-page audit fails the build on any such collision. The second pass is judgment: the normalizer catches spelling only, so a person or an agent then decides whether “function calling” and “tool calling” are one concept. A variant becomes a section on the existing page, never a second record. Mechanical rules catch spelling; only judgment catches meaning.