AI data engineering: the pipeline that makes everything else possible
Ingestion, cleaning, PII handling, chunking, embeddings and features, served from pgvector or Milvus and refreshed when sources change, so your RAG systems, agents and models answer from data you trust.
From raw sources to a serving layer the model can trust
Every answer a RAG system gives, every decision an agent makes, is downstream of this pipeline. The four stages are simple to name and easy to get subtly wrong.
The pipeline, with the loop most teams forget: when a source changes or a document is deleted, the index must change too, or the model keeps answering from the past.
Stage by stage
What each stage does, what we use, what goes wrong
Stage
What it does
Tools we use
Typical failure when skipped
Ingest
Connect to sources, parse formats, capture metadata and permissions, track versions
Python and Node.js connectors, Docker workers, Cloudflare R2 and Azure storage for raw copies
Permissions lost at the door, so retrieval later shows users documents they should not see
Clean
Remove duplicates and boilerplate, normalise text and tables, classify and redact personal data
Rule-based and model-based classifiers, redaction libraries, PostgreSQL or MySQL/MariaDB staging
Ten near-identical versions of a policy crowd out the right one; PII reaches the model provider
Chunk and embed
Split by structure, attach metadata, generate embeddings or features, version them
Embedding models chosen per our embeddings strategy; batch jobs with retries
Chunks cut mid-table or mid-clause; model changes silently invalidate the index
Serve
Store vectors and features, filter by permission and metadata, answer fast, expose a stable API
PostgreSQL + pgvector, Milvus, Cloudflare D1 for metadata, feature APIs in Node.js or Python
Slow queries at scale; no way to roll back a bad index
Refresh
Detect changes and deletions in sources, update or remove affected chunks, re-embed on model change
Change data capture, scheduled jobs, event triggers
Deleted contracts still answered from; last year's prices quoted as current
Who this is for
Teams whose AI is only as good as the data under it
Knowledge base and RAG owners
Your assistant answers from stale or duplicated content. The fix is upstream, in the pipeline feeding the AI knowledge base.
Predictive and analytics teams
Models need features computed the same way in training and in production. A feature pipeline with a serving API removes the drift.
Regulated organisations
You need lineage, permissions and retention to be provable. The pipeline is where that evidence is produced, reviewed with our AI data privacy practice.
Process
First source live in weeks, not quarters
Inventory
Sources, owners, formats, volumes, permissions, change rates, sensitivity. Prioritise by what the first use case needs.
Thin pipeline
One source end to end: ingest, clean, chunk, embed, serve. Downstream teams start building against it.
Harden and widen
Refresh loop, lineage, monitoring, more sources. Embedding model chosen with evals; see choosing embedding models.
Operate
Freshness and quality dashboards, cost per source, index versioning and rollback, quarterly review.
What goes wrong
Example: a knowledge base is loaded once from a shared drive. Six months later it confidently quotes a withdrawn policy because nobody built the refresh loop, and it shows a contractor an HR document because permissions were never captured. Neither is a model problem. Both are the pipeline.
Because the model only sees what the pipeline gives it. Duplicate documents, stale versions, missing permissions and badly chunked text produce confident wrong answers no prompt can fix. Our guide to data readiness for AI lists the checks we run first.
What is chunking and why does it matter?
Splitting documents into pieces small enough to embed and retrieve, while keeping each piece meaningful. Chunk boundaries decide what the model can see at answer time. We tune them per document type and test with the eval set. See chunking and embeddings.
Which vector database do you use?
PostgreSQL with pgvector when you already run Postgres and the scale fits; Milvus when you need dedicated vector infrastructure at larger scale. We also work with Pinecone, Qdrant and Weaviate. The choice is made with our vector database selection method.
How do you handle personal data in the pipeline?
Classify at ingestion, redact or tokenise before embedding where the use case allows, carry source permissions through to retrieval so users only see what they may, and set retention per source. Reviewed with our AI data privacy practice.
How long does it take to build a pipeline?
A first pipeline over two or three sources with embeddings and serving typically takes four to eight weeks. Adding change-data-capture, a feature store or many sources extends that. We ship the first source end to end early so the downstream team can start.
Ready to make AI work in production?
Tell us about your systems, your data and your constraints. We reply within one business day with a straight answer and a suggested next step.