The interesting engineering is not the scraping. It is everything
between a raw public record and something an agent should act on.
Public hiring and press sources are pulled in, then structured by a
language model rather than by brittle per-site parsers, because the
sources change layout constantly and a parser that breaks silently is
worse than one that never existed. Each record is then checked
against county property-ownership records, filled in from public
professional directories, and matched to a person, which is the step
that decides whether two similarly named records are one person or
two.
Only after that does anything become a ranked signal, and even then a
person reviews it before it is treated as a lead. That ordering is
deliberate. A signal that is wrong about who someone is does not
waste a minute of an agent's time; it costs them a relationship.
Under the hood: stack and testing
- Stack
- Python, Flask, Postgres, Typer, static HTML/CSS/JS
The test suite records real HTTP interactions as fixtures and replays
them, so the pipeline can be tested end to end without touching a
live source. It also means a source changing its markup shows up as a
failing test rather than as a quiet gap in the data.