Pagination is common technique used by web developers to display the large data set in search or list pages result instead loading the entire set of product on page load event. It's very easy to setup the pagination in Agenty using the Pagination feature with "Next page selector".
Find the Next button
Go to the web page you want to crawl and find the unique CSS selector of next button using the Chrome extension or manually by inspecting the element in chrome developer tool. For example, I used the chrome extension in below example and found
a.next as the unique selector for next button in this page.
Go to your scraping agent page and click on the
Edit agent button will take you to agent editor online as in screenshot below. Now, go to
Ajax & Pagination tab and enable the pagination and also paste the Next button selector in pagination section as detailed below for each option in the section :
- Enable pagination : True/False
- Pagination type : Click - This feature is for click pagination only and you may open a expert support ticket for any other advance pagination
- Next page selector : The unique CSS selector for Next button - The agent will click on that button to paginate until that button is visible and enabled
- Page limits : Maximum number of pages need to be paginated - The maximum number can be anything like 100 or 1000 but the agent pagination will exit if "Next" button not found or disabled after a number of clicks. So the web scraping with pagination will keep running until it reaches to pages limit or next button invisible/disabled on web page.
Once the Pagination configuration is completed,
save the agent and re-run to scrape the data from multiple pages automatically.