HTTP GET or
POST request to server and fetches the data; then the function parse the response and append the content on webpage.
To start with infinite scrolling web-pages scraping, the first step will be to keep developer mode (press F12 button) open in your Chrome or Firefox web browser. Then go to network tab, by default the browser will show your all request like images, CSS, fonts etc... but you can click on XHR button to filter AJAX requests only like I did in screenshot below.
Note : These event will execute only when you scroll down the page or click on load more button if any
Most likely you will see
POST requests there, coming through some internal web pages something like below
So once you found the actual back-end pages where the data is fetched from, why extract from html? Using Agenty we can extract data directly from these internal JSON or XML pages using
If it's a
GET method, simply append the query-string parameter in URL and enter in input url list to crawl.
If it's a
POST request, enter the encoded data need to be posted in a separate column and crawl with HTTP POST method.