How to deal with error 301 in scraping?

How to deal with error 301 when the response location is a only a part of the url needed?

I.E: I request a website https://www.site.com/users/page?=1

I get a response location "/users/page?=1"

How do you deal with that?

Posted by anonymous 1 years ago


You can enable auto redirect and set the maximum redirect value to let DSS know to follow the 301 redirect.

In order to do this, follow the steps below

  1. Edit the scraping agent
  2. Go to advance settings
  3. Click on "Throttling" tab
  4. Check "Enable auto redirect" and set the values.

auto redirect in scraping

Posted by anonymous 1 years ago

Topic Closed! This question is closed and don't accept posts now.

Close me