OSINT collectors run fast but produce noise. Existing systems scatter fragments across thousands of messy records—entity resolution requires manual triage, and the planning phase (question to queries) is underbaked. The cost of agentic crawling without governance is brutal: budgets evaporate, focus scatters. Vortex addresses the structural failure: it concentrates effort on the hard problems that scrapers dodge—the question planner and the entity-resolution reasoner—while enforcing deterministic governance at every stage.
Scrapers excel at fetching pages. They fail on two mechanisms that actually matter: the planner—converting a research question into the right 200 queries—and the reasoner—resolving 5,000 messy fragments into 30 confident entity profiles. Without governance, agentic crawlers wander. Off-allowlist queries multiply. Saturation kicks in. Cost climbs. The collector part is plumbing; the hard part is knowing what to collect and then assembling it into something real.
Vortex is a layered system: a Python engine orchestrates collection, ingestion, enrichment, faceting, synthesis, and rendering, while a deterministic Node governor—called the wall—gates every query and every fetched page. The governor enforces a tiered source allowlist, a dual-token relevance filter (both an ICP token and a work-signal token must be present), hard budget and saturation limits, and no off-allowlist wandering. Every fragment carries provenance metadata. Gemini handles extraction and synthesis at the collection and reasoning layers. The planner translates the research question into a focused query set. The reasoner ingests scattered fragments and produces clean entity profiles, each with its sourcing receipts visible. Darknet sources exist only for enrichment, never discovery.
One question in, governed collection out, clean entity profiles at the end.
The Python engine and Node governor are production-ready; wall configuration is complete and documented. All prospect data and entity profiles generated during development have been scrubbed—only synthetic sample data remains in the repo. The planning algorithm, entity-resolution logic, and complete provenance architecture are all versioned. The system is ready to run on new research objectives and produce end-to-end entity profiles with full sourcing receipts.
All real prospect lists, client names, and entity profiles generated during development have been removed. The repository contains only the engine code, the governor, configuration templates, and synthetic sample data. Actual scraping output is withheld—it comprises client work and competitive research. This is deliberate: the intellectual property is the system design itself, not the profiles it produces.