Throttling

Throttling features allow to add some (seconds) delay in your scraping agent. Because, the website scraping agent is blazing fast and can make multiple concurrent requests to make the most of the capability under your pricing plan and the power of cloud machines. However, with great power comes with great responsibility and the first-rule of website scraping is not to harm the target website. So, we can use the Throttling feature to add few seconds delay in sequential requests.

There are two types of Delay options :

  1. Fixed Delay
  2. Random Delay

Fixed Delay

The fix number of seconds to wait in each sequential requests

Before

If you see the screenshot below, the Throttling feature is not used in this agent, so there are no wait and the scraping agent is running continuously for all the inputs one-by-one.

throttling before

Steps to configure Throttling with a agent

Steps :

  1. Edit the agent by clicking on th Edit tab

  2. Go to Throttling section and Enable Request Throttling switch

  3. Select the Delay Type as 'Fixed delay' from drop-down

  4. Now enter the seconds to delay value

  5. Then Save the scraping agent configuration.

    fixed throttling function

  6. Finally, re-run your agent to apply the changes.

After

We can see the agent logs to ensure the agent is waiting for given interval time before making each next request, as in screenshot below:

throttling after

Random Delay

Random Delay option is used to add random number of seconds delay on each request. Agenty will automatically generate a random number between 0 and the value you enter in Seconds of delay option, and then use that number to wait between requests.

Steps

Follow the same steps as in Fixed delay, but select the Delay Type as 'Random delay' from drop-down as in screenshot below :

random throttling functin

After

Now, if you see the agent logs throttling event, it says waiting 7 seconds on first URL and then waiting 6 seconds for second URL. So, Agenty generated the random number between 0 and 10 and applied in Throttling configuration to delay in consecutive requests.

throttling after