Web Scraping with Screenshot

The scraping agent allow you to capture the full-page screenshot of page you are crawling, as it is very common in web scraping to capture a screenshot of a website for testing or as a proof of data being scraped is exactly same as it’s shown on the website.

There are 2 types of screenshot option available in Agenty

Full page screenshot

To capture the full page screenshot, follow these steps

  • Go to your agent page
  • Click on the edit tab and scroll down to Fields section
  • Add a new field with configuration like this
  • Name : Any name of your choice
  • Type : Default
  • From : PAGE_SCREENSHOT

full page screenshot with web scraping

screenshot in scraping result

So, if you are looking to capture the whole page screenshot with web scraping, use this option. Here is the example full page screenshot — https://www.pinterest.com/pin/486740672227404017

Visible page screenshot

This option allow you to capture the above the fold or visible part of website on computer screen without scrolling, similar to a normal screenshot your take using Print Scrn option in keyboard

To capture the visible portion screenshot of a webpage - We need to add a meta variable in the agent configuration, with key as screenshot.visible and value as true capture visible page screenshot

And save the agent configuration. Now when the scraping job is executed, Agenty will capture the visible page only screenshot as per the viewport width and height given in the web scraping engine.

Here is an example visible page screenshot captured in this agent with viewport : 1536x754 and JavaScript, images option enabled - https://www.pinterest.com/pin/486740672227480014