Description
Preparing web data for LLMs means writing crawlers, cleaning HTML and converting formats. WaterCrawl is a web app that turns web content into LLM-ready data, crawling pages and extracting what matters.
It is built on Python, Django, Scrapy and Celery and starts locally with Docker.
Crawling:Batch site capture.
Extraction:Cleaned into Markdown.
Queues:Async with Celery.
Docker:Quick local start.
It is built on Python, Django, Scrapy and Celery and starts locally with Docker.
Features
Crawling:Batch site capture.
Extraction:Cleaned into Markdown.
Queues:Async with Celery.
Docker:Quick local start.
