Skip to content

BigQuery

Append into a partitioned table, with the contract version carried as a column you can filter on.

Get API keys
Category: Warehouse

Results append into a date-partitioned table. The actor’s contract version rides along as a column, so a query can pin itself to one shape and stay correct across a wire bump.

That column is the whole trick. Without it, a contract change silently mixes two shapes in one table and every downstream query becomes a guess.

What it does

  • Appends via the Storage Write API — no staging files, no load jobs to babysit.
  • Partitions on run date, clusters on the actor’s natural key.
  • Adds _contract_version and _run_id to every row.

What you need

A dataset and a service account with roles/bigquery.dataEditor on it.

Ready to stop babysitting scrapers?

When no scraper exists for your source, BatScrape ships one.

Get API keys
  • One contract for every source
  • Versioned wire, no surprises