How does a data pipeline work?
Three stages repeat in almost every design. Ingestion pulls data from source systems such as an ERP, a CRM, files or APIs. Transformation applies data cleaning, joins, enrichment and data quality checks. Loading writes the result where it is needed. An orchestrator schedules each step, retries failures and records what ran. A pipeline is only as trustworthy as its checks between stages.
Batch vs streaming vs ETL vs ELT: what’s the difference?
| Pattern | How it moves data | Good for |
|---|---|---|
| Batch | In scheduled runs, for example nightly | Reports, model training sets |
| Streaming | Continuously, event by event | Fraud alerts, live dashboards |
| ETL | Transform, then load | Strict schemas, sensitive data cleaned before landing |
| ELT | Load raw, transform in the warehouse | Flexible analytics, cheap storage |
Pick the pattern from how fresh the answer must be, not from the tool you already own.
What does an AI data pipeline add?
A retrieval system needs a pipeline that ends in a vector database: fetch documents, strip boilerplate, split them into chunks, compute embeddings, and upsert them with metadata such as owner, date and access group. The refresh path matters as much as the first load, because an index that misses deletions keeps serving documents that no longer exist.
How do we make our data pipelines resumable?
Our scheduled pipelines that pull paid search data use the provider’s standard queue rather than its live endpoint, because queued tasks are cheaper: they post a batch of tasks, poll a readiness list every 30 seconds, and fetch only the results that list has named, since an early fetch returns an answer that looks empty but only means not ready. A task still missing after 20 minutes is fetched live and noted. Every run keeps a running cost total and stops before a call that would pass its budget. Long runs checkpoint per item and resume from the last completed one, so a failure costs one step, not the whole run. Not ready and empty must never look the same to the next stage.