A huge share of the knowledge your AI needs lives on the web: vendor docs, internal wikis, competitor pages, blog posts. Getting from a URL to something an agent can retrieve takes three steps — fetch, clean, index — and each has traps.
The traps
- JavaScript-rendered sites return an empty shell to plain HTTP fetchers — you need a real browser fallback.
- Raw HTML is mostly chrome: nav, ads, cookie banners. Main-content extraction decides your retrieval quality.
- One page is never enough — you want the whole docs section, crawled within limits.
The pipeline in Kit for AI
- Paste one URL or a whole list — multiple URLs paste straight into the ingest queue.
- Each page is fetched (with browser rendering when needed), stripped to its main content, and converted to clean Markdown.
- Crawl mode follows same-site links up to your page budget.
- Everything lands in a knowledge base with hybrid search and cited chat, reachable over API and MCP.
Point it at a docs site you rely on and ask a real question against the result. That's the fastest way to judge any ingestion pipeline — free to try.