High-performance concurrent web crawler with real-time dashboard, AI-powered content analysis, and intelligent filtering.
Anyone can write a script that fetches a few pages. The difficulty shows up at the fourth domain and the six-hundredth page — when you need to stop and resume without losing progress, avoid re-fetching what you already have, and still see what the thing is doing.
The Go backend is the engine. A configurable worker pool processes pages in parallel off a priority queue, converts HTML to Markdown, and writes the result with JSON metadata organised by domain. Caching is time-based and self-invalidating, crawls resume where they stopped, and robots.txt and polite rate limiting are honoured rather than optional.
The dashboard in the shots below is running against eight domains and 679 stored pages, with 1,781 URLs still queued. The per-domain view goes further: 616 pages from developers.pismo.io, with an active five-rule filter set stripping ReadMe meta tags, CDN resources and cookie banners before anything is stored — the difference between a crawl and an archive worth reading.
React and TypeScript on the front, Go modules split by concern behind it — API handlers, cache, HTML-to-Markdown, crawler, priority queue, filesystem store. Configuration comes from environment variables, a JSON file or the UI. The whole thing runs in Docker.