The first instinct when you need to scrape four sites is to write one scraper with four configs. It is the right instinct and it is wrong, and it takes about six months to find out which.
How it goes
Source one and source two are similar enough that the config approach feels vindicated. Source three paginates differently, so you add a pagination strategy. Source four rate-limits by IP rather than by key, so you add a throttle mode. Source five returns prices as strings with currency symbols, so you add a normaliser hook.
By source eight, the generic scraper is six scrapers sharing a namespace, and a change to the pagination strategy for source three can break source seven. You have all the coupling of a monolith and none of the benefit.
The alternative
One actor per source. Each is its own repository, with its own dependencies, its own release cadence and its own failure surface.
What they share is not code. It is the contract — vendored in, version-pinned, and gated at build time. Sources are wildly different; their output is not, because output shape is the one thing that is genuinely common.
What you give up
Duplication, and you should be honest about it. Two actors scraping two sites that happen to run the same CMS will contain similar code, and that will itch.
Resist it for a while. The similarity is usually coincidental, and factoring it out couples two things whose only real relationship was that they looked alike in month two.
What you get
A broken actor breaks one source. A dependency bump touches one actor. A new engineer reads one repository. And the blast radius of a mistake is exactly as large as the mistake.
Ada Okafor
Head of Data
