Before writing a single selector, spend twenty minutes finding out whether you need selectors at all.
Most sites that render a list of things fetch that list as JSON and then paint it. If you can find the fetch, you get structured data, stable field names and a scraper that survives a redesign. If you parse the painted result, you get none of those.
Where to look, in order
1. The network tab, filtered to XHR/fetch. Scroll the page, watch what loads. A paginated list almost always reveals its endpoint on page two.
2. The initial HTML. Many frameworks inline the first payload for hydration:
search the raw response for __NEXT_DATA__, __NUXT__, self.__next_f, or just
a suspiciously large <script type="application/json">.
3. The bundle. If the endpoint is constructed rather than literal, the URL template is in the JavaScript. Pretty-print it and search for the path fragment you saw in the network tab. Minification mangles identifiers, not string literals — the URL survives intact.
4. Only then, the DOM.
What to check before you commit
- Does the endpoint need the page’s cookies, or will it answer cold? Cold is a much simpler actor.
- Is there a cursor, or only offset pagination? Offset pagination over a changing dataset silently skips rows.
- Does it return more fields than the page shows? It usually does, and some of them are the ones you actually wanted.
The honest caveat
Sometimes there is no endpoint, the markup is server-rendered, and you are parsing HTML like it is 2009. That is fine. Just make sure it is true before you accept it.
Priya Nandakumar
Founding Engineer
