Ingestion is the unglamorous half of every RAG system: getting real-world content — PDFs, Word files, spreadsheets, scans, web pages — into a clean, retrievable form. Do it badly and no amount of prompt engineering saves you; the model retrieves noise.
What a good ingestion pipeline does
- Converts any format to clean, token-efficient Markdown — structure preserved, boilerplate stripped.
- Handles the hard inputs: OCR for scans and images, real browser rendering for JavaScript-built pages.
- Ingests in batch — paste a list of URLs or drop a folder of files, not one upload at a time.
- Lands everything in a knowledge base where retrieval is built in, not a separate project.
Ingesting with Kit for AI
Paste multiple URLs (Ctrl+V works — they queue automatically), drop files, pick a knowledge base, and ingest. Each item is converted to Markdown, chunked, embedded, and made searchable. The same flow is one API call for pipelines, and agents can do it themselves over MCP.
The details that bite later
- Scanned PDFs and images need OCR — silent empty conversions poison retrieval.
- JS-rendered pages return empty shells to plain fetchers; you need a rendering fallback.
- Re-ingesting the same content should dedupe via caching, not double your storage.
Kit for AI handles all three by default. Ingest something real and search it — free to try.