Cursor state belongs in the core, not in your job queue

Here is a failure everyone who has run a scraper at scale has met.

A run dies two-thirds of the way through. Your job queue retries it, because that is what job queues do. The actor starts from the beginning, re-fetches everything it already fetched, and you pay for all of it twice.

The usual fix is to make the job idempotent by tracking progress somewhere — a Redis key, a row in a jobs table, a file in a bucket. It works. It also has to be reimplemented, slightly differently, by every actor.

Making it structural

We moved cursor state into the core, alongside auth, metering and quota. An actor does not decide how to resume; it is handed a cursor and told where to start.

That has a few consequences worth naming:

• A retry is free. The run resumes at the last committed cursor. The rows before it are not re-fetched and not re-billed.
• The cursor commits with the rows. Not before, not after. If the write fails, the cursor did not move, and the next run redoes exactly the work that was lost.
• Actors get it for free. A new actor is resumable on the day it is written, because resumability is not something it implements.

The part that is actually hard

Committing the cursor with the rows means the destination write and the cursor update have to land together. For Postgres that is one transaction and it is easy. For an object store it is not, so the cursor goes into the run manifest and the manifest is the last thing written — a run without a manifest is a run that did not finish, and the next one starts from the previous manifest.

It is not elegant. It is correct, which matters more.

Why this is a core concern

Because the alternative is asking every actor author to be careful about billing. People are careful right up until the day they are shipping something else, and then the careful part is the part that gets skipped.