How to deal with HTTP 301 redirect error in web scraping?

HTTP 301 redirect error means that, the URL we are using in our scraping agent input is old or expired. And the web server is permanently redirecting to another URL. The server returns the response code as 301 and message 301 Moved Permanently to tell the clients to immediately retry the alternate URL.

We can configure our scraping agent, by enabling the auto redirect option. And can, also set the maximum redirects value in number to tell Agenty, if redirect should be followed and how many times (if yes). 

Steps

  1. Edit the scraping agent
  2. Advance options
  3. Go to Auto Redirect Handling section
  4. Switch on the Enable auto redirect option

    error 301
     
  5. Set the maximum redirect value, by default : 3

The Auto-redirect option is by default enabled in all newly created scraping agents after July, 2018