BigQuery
Append into a partitioned table, with the contract version carried as a column you can filter on.
Results append into a date-partitioned table. The actor’s contract version rides along as a column, so a query can pin itself to one shape and stay correct across a wire bump.
That column is the whole trick. Without it, a contract change silently mixes two shapes in one table and every downstream query becomes a guess.
What it does
- Appends via the Storage Write API — no staging files, no load jobs to babysit.
- Partitions on run date, clusters on the actor’s natural key.
- Adds
_contract_versionand_run_idto every row.
What you need
A dataset and a service account with roles/bigquery.dataEditor on it.
Ready to stop babysitting scrapers?
When no scraper exists for your source, BatScrape ships one.
Get API keys
- One contract for every source
- Versioned wire, no surprises