How can I have the scraped URL appear in the output?

In order to get the current or scraped URL in output table, you can simply add a default field REQUEST_URL or RESPONSE_URL in your scraping agent collection.

  • REQUEST_URL : The request URL is same as it was in input.
  • RESPONSE_URL : The response URL is served by the web server. For example, if your input URL is redirecting to some other URL, the response URL will give you the redirected URL while the request URL will display the same URL as it was in input.

Step 1 : Edit the agent - 

Click on the edit button to edit the scraping agent

edit scraper

Step 2 : Add the default field - 

add current url to scraper

Step 3 : Save the agent - 

Now, the field is added to agent configuration and the value will be populated next time the agent is executed manually or via scheduled jobs.