Xandrov Labs built the Agentic Crawler to test a fundamental question before committing to architecture: could three free components—automated URL discovery, content extraction, and a single LLM inference pass—produce useful results on their own?
The system remains published unpolished, a deliberate lineage artifact. It documents the lab's methodology: keep experiments and their failures. Their limits become the specification for the next system. The constraints exposed here—hallucination, identity conflation, rate-limit bursts—are not flaws that would be fixed. They are the problems that motivated the two-layer redesign.
The fundamental question: could a three-stage pipeline using free tools at each stage produce useful output? The stages were discovery (finding candidate URLs), extraction (cleaning content from those pages), and inference (one LLM pass over the extracted data).
The architecture was minimal by design—one Python file, no queuing, no validation layer, no error recovery. This meant hallucination in the model output had nowhere to be caught. Identity conflation happened silently. Rate-limit bursts from extraction would simply fail, cascading through the pipeline.
The system was one Python script implementing three stages. Discovery used an automated search or crawling technique to generate candidate URLs—the mechanism is not detailed here. Extraction took those URLs and cleaned the HTML content, reducing it to plain text suitable for processing. Inference ran the extracted content through a single LLM call, asking it to identify patterns or answers in the aggregated data.
Because everything was in one file with no intermediate storage, the pipeline was fast but rigid. Data flowed discovery → extraction → inference with no buffering. The extraction stage used free or public APIs. The model call was also single-pass: no iterative refinement, no verification loop, no way to flag uncertainty.
The minimal design was intentional. This was a proof-of-concept, not a production system. It was meant to expose exactly where free tools break: where hallucination happens, where identity conflation occurs, where rate limits become blocking issues.
Experiments are kept. Their limits become the spec for the next build.
The Agentic Crawler proved the core hypothesis: three free stages could work together. It also proved the limits. Hallucination in LLM output went uncaught. Identity conflation accumulated silently. Rate-limit bursts became a blocker. These constraints became the specification for the two-layer redesign. The Agentic Crawler itself is superseded but kept in the lineage as a reference point—it shows where the lab started.
The dossier omits the discovery mechanism's source—whether it was a search API, web crawler, or seed URL list. The extracted HTML samples and LLM outputs are not included; these would include hallucinated entities, conflated identities, and patterns that don't exist in the source data. The specific API keys and extraction libraries are also withheld. What remains is the architecture and decision logic.