Skip to content
batscrape

Amazon S3

Land raw and normalised output in your own bucket, partitioned by run and contract version.

Get API keys
Category: Storage

Every run writes two things to your bucket: the normalised rows, and the raw payload they were derived from.

Keeping the raw payload is the part people skip and then regret. When a normalisation bug surfaces three weeks later, the raw copy is the difference between re-deriving the data and re-scraping the source.

Layout

s3://your-bucket/
  <actor>/v<contract>/<run-id>/
    rows.jsonl.zst
    raw/
    manifest.json

manifest.json carries the contract version, the cursor the run started and finished at, and the row count — enough to reconcile a run without opening it.

What you need

A bucket and an IAM role with s3:PutObject scoped to one prefix. BatScrape never asks for list or delete.

Ready to stop babysitting scrapers?

When no scraper exists for your source, BatScrape ships one.

Get API keys
  • One contract for every source
  • Versioned wire, no surprises