Frequently Asked Questions

Here are some frequently asked questions to quickly get the answer.

What is the scraping agent?

The scraping agent is a set of configurations like fields, selectors, URLs etc. of a particular website scraping. Since, most of the websites have their own different structure so a single scraping agent can extract the data from a particular website only where it was setup. But can extract any number of pages with similar structure.

The scraping agent can be created using our Chrome extension available in Chrome store

How does page credit work?

Agenty counts the page credit on each successful (2xx) requests, so the pages credit will be deducted based on the number of requests you are sending.

  • 1 page credit = 1 input request

Here are some examples

  1. I have 15 URLs in my agent input and those are some list/search pages and contain 20 items per page. So, when the scraping job gets completed it will return around 300 records in output. How many pages credit will be deducted?
    Ans: 15 pages credit
  2. I have 20 different URLs of same website and each has to paginate to crawl the next available pages. Let's say 5 more pages each and those are list/search page with fixed 50 products each page. So at the end of scraping there will be around 6000 output records. How many page credit will be deducted?
    Ans: 20 + (20*5) = 120
  3. I have a list of 100 product details pages and each URL is going to extract 40 fields like (product name, url, price, description etc) for each product. When the scraping agent completed I saw 2 URL failed out of 100 as those were invalid and website returned 4xx error for them. How many pages credit will be deducted in this case?
    Ans: 98 pages credit (Failed request not counted in page credit)

What is a successful request?

A successful request on web return 2xx HTTP status code. Agenty consider them as a successful request and counts that in page credit.

How can I delete unused agents?

You may delete agents by visiting the agents page in your Agenty account > Then, select the agents by clicking on the check box and then hit the Delete button

delete agents

How do I cancel my subscription?

If you've subscribed for Agenty plan and subscription is set to automatically renew on monthly basis you can cancel your subscription from your account page. To turn off the automatic renewal, login and then

  1. Go to your Setting > Subscription
  2. And click on the Cancel Subscription button as shown in screenshot.

subscription

Your account will remain active until your renewal date and after that you will no longer be charged and the account will be suspended.

Please note: when you hit the Cancel Subscription button it doesn't process immediately, instead the request is submitted with billing team to cancel the subscription which might take up to 24 hours to reflect the change in your account and you will receive a confirmation email as soon the cancellation process is completed.

How do I delete my Agenty account?

Here are the steps to delete your Agenty account (everything) permanently from Agenty Systems.

  1. Go to Settings from left menu and then scroll down
  2. You would see the Delete account section, click on it
  3. A dialog box will open, click on Delete my account button to delete your account

Note: Deleting account is an irreversible operation. Any data synced to your account will be deleted and no longer be accessible. Please ensure that you've signed out of all Agenty instances before you delete your account.

deleteaccount2

How to scrape data behind a link?

To scrape data behind the link, you need to create 2 scraping agents and connect them with each other to go to details page :

  1. The first scraper will scrape data from list page including DETAILS_PAGE_URL.
  2. Then create the 2nd scraping agent for details page and point it to first agent result using URL from source agent option in input.

See this complete documentation with an example Scraping Details Pages from Listings.

How to scrape websites with captcha?

Scraping CAPTCHA websites is not supported in self-services plans, but we provide Managed Web scraping services with hybrid technology to bypass the CAPTCHA using human labor combined with bot for decoding CAPTCHA images and continue the crawler to get the potential data you need for your business. Get in touch with sales to request a quote.

This is how it works :-)

captcha website scraping

How to modify a web scraping agent?

Go to the agent details page and then click on the Edit tab, it will open the agent editor where you can change any setting you want in the agent configuration. edit web scraping agent

How do I download the agent result in CSV format?

Go to your agent page > click on the Export drop down button on the right corner of your result table > then CSV option to download the agent result in CSV format export agent result in csv format