Pagination - Click on the "Next" button to scrape multiple pages

Pagination is common technique used by web developers to display the large data set in search or list pages result instead loading the entire set of product on page load event. It's very easy to setup the pagination in Agenty using the pagination feature with "Next page selector".

Find the Next button

Go to the web page you want to crawl and find the unique CSS selector of next button using the chrome extension or manually by inspecting the element in chrome developer tool. For example, I used the chrome extension in below example and found a.next as the unique selector for next button in this page.

Configure Pagination

Go to your scraping agent page and click on the "Edit agent" button will take you to agent editor online as in screenshot below. Now, go to "Ajax & Pagination" tab and enable the pagination and also paste the next button selector in pagination section as detailed below for each option in the section : 

  • Enable pagination : True/False
  • Pagination type : Click  - This feature is for click pagination only and you may open a expert support ticket for any other advance pagination
  • Next page selector : The unique CSS selector for next button - The agent will click on that button to paginate until that button is visible and enabled
  • Page limits : Maximum number of pages needs to be paginated - The maximum number can be anything like 100 or 1000 but the agent pagination will exit if "Next" button not found or disabled after a number of clicks. So the web scraping with pagination will keep running until it reaches to pages limit or next button invisible/disabled on web page.

Note : Be sure to enable "Ajax & JavaScript Load" option if the pagination require the AJAX or JavaScript to be loaded in "Fetch pages via browser" section and disable if not required. By enabling the AJAX the agent will load and execute the entire scripts which might slow down your scraper.

web scraping pagination

Re-run Agent

Once the pagination configuration is completed, save the agent and re-run to scrape the data from multiple pages automatically.