Throttling features allow to add some (seconds) delay in your scraping agent. Because, the website scraping agent is blazing fast and can make multiple concurrent requests to make the most of the capability under your pricing plan and the power of cloud machines. However, with great power comes with great responsibility and the first-rule of website scraping is not to harm the target website. So, we can use the Throttling feature to add few seconds delay in sequential requests.
There are two types of Delay options :
- Fixed Delay
- Random Delay
The fix number of seconds to wait in each sequential requests
If you see the screenshot below, the
Throttling feature is not used in this agent, so there are no wait and the scraping agent is running continuously for all the inputs one-by-one.
Steps to configure
Throttling with a agent
Edit the agent by clicking on th
Throttlingsection and Enable Request Throttling switch
Select the Delay Type as 'Fixed delay' from drop-down
Now enter the seconds to delay value
Savethe scraping agent configuration.
Finally, re-run your agent to apply the changes.
We can see the agent logs to ensure the agent is waiting for given interval time before making each next request, as in screenshot below:
Random Delay option is used to add random number of seconds delay on each request. Agenty will automatically generate a random number between
0 and the value you enter in Seconds of delay option, and then use that number to wait between requests.
Follow the same steps as in
Fixed delay, but select the Delay Type as 'Random delay' from drop-down as in screenshot below :
Now, if you see the agent logs throttling event, it says
waiting 7 seconds on first URL and then
waiting 6 seconds for second URL. So, Agenty generated the random number between
10 and applied in Throttling configuration to delay in consecutive requests.