Defensive-security research often requires collection across multiple dark-web networks — Tor, I2P, ZeroNet, Freenet — and hybrid surfaces like tor2web. This work sits at the edge of infrastructure: collection is necessary, but so is operational discipline. The stack exists as part of a larger OSINT and threat-intelligence toolchain within the lab.
Paired elsewhere with a custom darknet MCP server and threat-intel feed aggregation, the crawler is one layer of a research-grade intelligence pipeline. The focus is deliberate: orchestration and configuration, not engine-building.
Dark-web collection is fundamentally an operations problem. Running crawlers across multiple networks means managing several concurrent daemons — one per protocol — each with its own quirks, seed requirements, and rate-limiting expectations. Add Redis workers for queue management, tuning for multiprocessing, broker configuration, and the complexity compounds quickly.
Building a crawler engine from scratch is tempting but inefficient. The right move is to configure a proven, maintained upstream engine well: handle deployment, orchestration, and tuning, leaving the core crawler logic to those who built and maintain it.
The deployment is orchestrated through Docker Compose, the standard tool for managing multi-service systems. The crawler engine is JarryShaw's darc — an open-source, actively maintained Tor/I2P/ZeroNet crawler — deployed as a containerized service alongside a Redis instance for work-queue management.
Configuration happens at three layers. First: broker setup for darc itself — defining crawl parameters, rate limits, and network-specific tuning. Second: multiprocessing configuration to balance parallelism against resource constraints, a critical tuning point for sustained collection at scale. Third: per-network seed lists — the entry points and known addresses for each protocol, sourced from research-grade OSINT feeds.
The architecture is deliberately minimal: it does not extend darc, does not fork it, and does not rewire its internals. It is a thin orchestration layer — Docker, Redis, configuration — on top of a respected upstream project. That constraint keeps maintenance burden low and updates feasible.
Configure the proven engine; publish the knowledge, not the data.
The stack is production-ready for research-scale collection: it runs, survives deployment restarts, and scales horizontally via Redis. It is explicitly framed as defensive-security research and an orchestration layer over a credited upstream engine. The knowledge of how to deploy and tune this infrastructure is published; the collected data itself is not. That distinction preserves utility while maintaining ethical and legal ground.
All scraped output — the actual darknet content collected during research — is stripped from publication. Deployment patterns, configuration templates, and infrastructure tuning are preserved and useful; raw collected data is not. This choice reflects the lab's commitment to publish research methodology without publishing the raw intelligence itself.