Scrape the sites that need a real browser

Sometimes you grab a page and almost nothing's there. No list, no content, just a loading spinner. That's a page that builds itself a half-second after it opens, so a plain grab catches it before the content shows up. The fix is to use a real browser. The technical name is a headless browser: a normal browser running invisibly in the background, with Claude at the wheel.

You can have Claude drive an actual browser in the background. It opens the page, waits for everything to appear, scrolls, clicks "show more," and then you read the finished page, exactly what a person would see. The tool for this is called Playwright.

The trigger is simple

If a plain grab comes back empty, the content is being built on the fly, and you need a real browser to let it finish. That's the only signal you need.

Open it, and wait for the good part

The browser opens the page in the background (no window pops up). The one mistake everyone makes is reading the page too soon, before it's done loading. The fix is to wait for the actual content to appear instead of waiting a fixed number of seconds.

Scroll and click for the rest

Lots of sites only show the first handful, then load more as you scroll or when you click a button. A real browser can do both. Scroll to the bottom, click "show more" until it's gone, and the page keeps filling in. Once it stops growing, you've got it all.

Then it's the workhorse again

Once the page is fully loaded, you're right back to the everyday play: hand the finished page to Claude, ask for the parts you want, get a clean list. The browser's only job is to turn an empty-looking page into a full one.

A real browser is heavier and slower than a plain grab, so it's the second choice, not the first. Try the simple grab first, fall back to the browser only when the page comes up empty, and most of your work stays fast and free.