Run BM25 search on llms.txt indexes and add sources at runtime.
Fetch webpages, extract clean content, discover links, and batch process URLs.
