Lead crawlers drive outreach pipelines — they scrape the public web for high-intent freelance and contract postings, identify who posted them, gather contact channels from those identities, and emit a structured outreach record.
Two failure modes kill the entire pipeline. Hallucinated extraction: the model finds a lead that isn't there. Identity conflation: two people who share a name become one record, collapsing different contact channels and jobs into a single phantom identity. Free-tier API limits punish concurrent requests on top, forcing a choice between speed and quota.
Hallucinated extraction kills crawlers — the model finds a lead that isn't there, producing a phantom outreach record you act on anyway. Identity conflation is worse: two people who share a name collapse into one record, mixing contact channels and job context irreversibly.
Free-tier rate limits punish naive concurrency, forcing a choice between crawl speed and staying in quota. Standard mitigations — higher API spend, broader concurrency, naive deduplication — all trade infrastructure cost for accuracy. The real bottleneck is verification, not discovery.
Discovery runs through DuckDuckGo via the ddgs library. Raw HTML is cleaned to markdown through Jina Reader. Layer 1 applies strict extraction under a pydantic schema — bad JSON triggers one re-request, then the record is dropped; low-intent postings are dropped immediately.
A deep sub-crawl gathers identity context from the poster's digital presence: prior work, contact channels, professional patterns. Layer 2 runs as a fresh, isolated synthesis call with no access to Layer 1's intermediate states or findings; this enforces verification over amplification.
The system hard-prompts Layer 2 never to connect two people just because they share a name. An asyncio orchestrator chains the whole pipeline and paces requests to stay rate-safe on free tier.
Layer 2 runs isolated: verification over amplification, no identity conflation, free-tier safe.
The system is a working pipeline with its limits documented in the open — bot walls, free-tier daily caps, edge cases in identity synthesis. It grew from a single-file seed crawler experiment and is now modular enough to integrate into larger discovery workflows. The trade-off is clear: zero infrastructure cost in exchange for handling within free-tier quotas and acknowledging where bot detection will stop the crawler.
Live prospect lists and scraped output are withheld — the crawler produces working records with real contact channels, and publishing examples would hand over a ready-made outreach list. What stays is the pipeline itself: the two-layer architecture, the per-key rate governance, the extraction schema. The records it produces are not part of what's shown.