How to scrape multiple pages with a Chrome extension (pagination, infinite scroll, load more)
Most scraping goes wrong at page two. Here is how to get every page into one spreadsheet, whether the site uses Next links, a Load more button or infinite scroll.
Scraping one page is easy. The trouble starts when the list you want runs across 20 pages, or loads more items as you scroll, and your extension hands you the first 25 rows and stops. Search results, product listings, directories and job boards almost all work this way, so getting multiple pages right is most of the job.
This guide explains how to scrape multiple pages with a Chrome extension: the three ways sites split results (numbered Next links, Load more buttons, infinite scroll), the URL-parameter trick, how the free Grabby free web scraper Chrome extension handles each one, and what to check when rows go missing or repeat.
The three ways websites split results across pages
Before you scrape, look at what happens when you reach the bottom of the list. There are really only three behaviours, and the right setting depends on which one you are looking at.
- Numbered pages or a Next link (pagination). You click and a new set of results replaces the old one. Sometimes the whole page reloads and the address changes; sometimes the list swaps in place without a reload.
- A Load more button. You click and extra results are added underneath the ones you already have. The address usually stays the same.
- Infinite scroll. Nothing to click. New results appear as you scroll toward the bottom.
Which setting to use: a decision table
| What you see | Pattern | Grabby setting | What to point at |
|---|---|---|---|
| Page numbers, or a Next link or arrow at the bottom | Pagination | Pagination | The Next button or link |
| A “Load more” or “Show more results” button | Load more | Load More | The Load more button |
| No button, list grows as you scroll | Infinite scroll | Auto-Scroll | Nothing to click |
| Address ends in ?page=2, /page/2 or &start=20 | URL parameter | Pagination (follow the Next link) | The Next link, same as above |
| Only the current page matters | None | Just this page | Nothing |
How to scrape multiple pages with Grabby
Step 1: Select the list on the first page
Open the first page of results, click the Grabby icon, choose Scrape List and click one item in the list. Grabby highlights the whole repeating set and builds the columns for you. If you are new to this part, our guide to scraping a table or list from a website covers it in more detail.
Step 2: Choose how to load more
Grabby offers four options: Just this page, Auto-Scroll, Pagination and Load More. It also flags what it detects on the page (“pagination detected” or “auto-scroll detected”), which is usually the right choice. For Pagination and Load More you can click Pick on page and then click the exact button yourself. Do that whenever detection picks the wrong control, such as a “Next article” link in a footer.
Step 3: Set how far to go and start
For Pagination you set Max pages; for Auto-Scroll you set Max scrolls, which is the number of scroll passes. A run also ends on its own when the Next button disappears, when scrolling stops producing new items, or when you press stop. Start small, say three pages, to confirm the columns look right, then run the full thing.
Step 4: Export once
Every page lands in the same table, so you export one CSV or Excel file rather than stitching pieces together. From there, the Excel export workflow covers cleanup.
How a point-and-click extension handles each pattern
Numbered Next links
Grabby reads the current page, clicks the Next control, waits for the content to change (a new first item, more items, or a new URL), and repeats. If the site reloads the whole page, the run picks up again after the reload. If the site swaps content in place, it carries on straight away. It stops when it cannot find a Next control or the list stops changing.
Load more buttons
Load More works the same way, but the click adds rows to the page you are already on. The usual mistake is scraping before the extra rows have loaded, so give a slow site a moment and confirm the row counter keeps climbing.
Infinite scroll
Auto-Scroll keeps scrolling the page, including pages where the list lives in its own scrolling container rather than the main window, and reads the rows as they appear. It stops after two quiet rounds with no new items. Because it collects rows on every pass, lists that drop off-screen rows from the page as you scroll (so-called virtualised lists) still end up complete.
URL-parameter pagination
Many sites put the page in the address: ?page=2, /page/2/, or an offset such as &start=20. You could generate those addresses and open each one, but following the real Next link gets the same result with less setup, and it keeps working if the site changes its numbering. The address pattern is mainly useful for understanding what the site is doing, and for opening a specific later page to start from.
Why a multi-page scrape stops early or returns gaps
| Symptom | Likely cause | What to try |
|---|---|---|
| Stops after page 1 | Next button not found, or the wrong one was detected | Use Pick on page and click the real Next control |
| Rows have blank images or prices | Lazy loading: content appears only when scrolled into view | Keep Accurate scrape on, which scrolls each item into view before reading it |
| The same rows keep coming back | Sponsored or pinned items repeat on every page | Dedupe on the item link or ID after export |
| Clicks land on the wrong thing | A sticky header or cookie banner covers the button | Dismiss the banner first, then start; re-pick the button if needed |
| Fewer rows than the site claims | Result count includes items the page never serves, or the site caps visible results | Narrow the search with filters and scrape several smaller slices |
| CAPTCHA or error pages mid-run | Rate limiting from too many fast requests | Stop, wait, lower the page count, and continue later |
Sticky headers and overlays
A fixed header or a newsletter pop-up can sit on top of the Next button, so a click lands on the overlay instead. Close pop-ups before you start and watch the first page turn over. If it does not move, pick the button again.
Lazy loading
Images and sometimes prices only load once an item scrolls into view. Grabby’s Accurate scrape setting is on by default for this reason: it brings each item into view and waits for its content before reading it. Turn it off only if a site is simple and you want speed.
Duplicate rows
Grabby removes exact repeats as it collects, but near-repeats survive: the same product with a slightly different title, or a sponsored card that returns on each page. Sort by the link column and remove duplicates on that, not on the name.
Be polite when you scrape many pages
- Scrape what you need, not everything. Filter the search first so there are fewer pages to walk.
- Run large jobs when the site is quiet, and stop if you hit a CAPTCHA rather than trying to push past it.
- Do not hammer the same site repeatedly. Save your export and reuse it.
- Check the site’s terms of use and robots.txt. Terms vary, and some sites forbid automated collection outright.
Because Grabby runs in your own browser, it reads pages the way you do and at a human-ish pace, but that does not exempt you from a site’s rules. This is general information, not legal advice.
What to read next
Once pagination works, the rest is just choosing the right fields. See scraping a website to Excel without code, or, if you are coming from another tool, the free Instant Data Scraper alternative roundup. Pagination is also the step where most guides here get stuck, so it is worth getting this one right.
Frequently asked questions
How do I scrape data from multiple pages with a Chrome extension? Select the list on page one, then tell the extension how the site moves to more results: a Next link, a Load more button, or scrolling. Grabby offers Pagination, Load More and Auto-Scroll for exactly these cases, collects each page into one table and lets you export a single CSV or Excel file.
What is the difference between pagination, Load more and infinite scroll? Pagination swaps in a new page of results when you click Next or a page number. Load more appends results to the same page when you click a button. Infinite scroll appends results automatically as you scroll down. Each needs a different action from the scraper.
Why does my scraper stop after the first page? Usually it was never told how to move on. Pick the Next or Load more button yourself instead of relying on detection, check that the button is not disabled on the last page, and make sure the rows are still in the page when you start.
Why do I get duplicate rows when scraping several pages? Sponsored or pinned items repeat on every page, and some sites re-show the last few results when a new batch loads. Dedupe on a stable value such as the item link or ID, not on the title.
Can I scrape pages that change only a number in the URL? Yes. If the address changes like ?page=2, you can scrape each page by following the Next link, or by generating the URLs yourself. Following the real Next link is usually simpler and stays in step with the site.
Will scraping many pages get me blocked? It can trigger rate limits or CAPTCHAs on some sites, especially with big, fast runs. Keep page counts sensible, run during quiet hours, and stop if the site starts asking you to prove you are human. Also read the site’s terms first.
Grabby is free while in early access, with every feature unlocked and no account needed. Add it to Chrome and scrape every page of your next list.