crawler.sh

is a local-first tool designed to extract content from any website and convert it into clean, structured Markdown optimized for AI workflows. Unlike cloud-based services that charge per page or impose API quotas, runs entirely on your laptop, processing pages without sending data to external servers. It uses a custom JavaScript rendering engine to handle single-page applications (SPAs) and dynamic content that typical static scrapers cannot capture. The tool respects robots.txt directives, ensuring compliance with site policies, and does not require headless Chrome or any browser dependency. Each extracted page produces Markdown with minimal noise—only the main article content is retained, stripping away navigation bars, footers, ads, and sidebars. The output includes metadata such as word count, author byline, publication language, and an excerpt, making it straightforward to filter and organize large collections of documents. Users can bulk export an entire website as a single Markdown archive, preserving relative links and folder structure, which is useful for building offline knowledge bases or training corpora. The primary use case is preparing data for retrieval-augmented generation (RAG) systems, fine-tuning large language models, or feeding context into AI agents. For example, a developer building a RAG pipeline for customer support documentation can point at the help center’s URL, let it crawl all pages respecting robots.txt, and directly ingest the resulting Markdown into a vector store. The tool also generates a word count and excerpt for each page, enabling quick quality checks before training. includes a built-in MCP (Model Context Protocol) server, allowing AI agents to request content from specific URLs on the fly. This feature is useful for agents that n