Here are some frequently asked questions to quickly get the answer.
What is the scraping agent?
The scraping agent is a set of configurations like fields, selectors, URLs etc. of a particular website scraping. Since, most of the websites have their own different structure so a single scraping agent can extract the data from a particular website only where it was setup. But can extract any number of pages with similar structure.
The scraping agent can be created using our Chrome extension available in Chrome store
How does page credit work?
Agenty counts the page credit on each successful (
2xx) requests, so the pages credit will be deducted based on the number of requests you are sending.
- 1 page credit = 1 input request
Here are some examples
- I have 15 URLs in my agent input and those are some list/search pages and contain 20 items per page. So, when the scraping job gets completed it will return around 300 records in output. How many pages credit will be deducted?
Ans: 15 pages credit
- I have 20 different URLs of same website and each has to paginate to crawl the next available pages. Let's say 5 more pages each and those are list/search page with fixed 50 products each page. So at the end of scraping there will be around 6000 output records. How many page credit will be deducted?
Ans: 20 + (20*5) = 120
- I have a list of 100 product details pages and each URL is going to extract 40 fields like (product name, url, price, description etc) for each product. When the scraping agent completed I saw 2 URL failed out of 100 as those were invalid and website returned 4xx error for them. How many pages credit will be deducted in this case?
Ans: 98 pages credit (Failed request not counted in page credit)
What is a successful request?
A successful request on web return 2xx HTTP status code. Agenty consider them as a successful request and counts that in page credit.
How to scrape a website?
You'd need to create a scraping agent to scrape data from a website.
- Install our Chrome extension
- Go to the website page you want to scrape
- Launch the extension to setup a scraper - See this step-by-step tutorial to learn how to create a scraping agent
What is the limitation in FREE plan?
The Free plan is limited to process 100 pages, or export 100 rows only (whichever comes first) using any agent (Web Scraping, Textract, OCR etc.) available in Agenty.
How to scrape data behind a link?
To scrape data behind the link, you need to create 2 scraping agents and connect them with each other to go to details page :
- The first scraper will scrape data from list page including DETAILS_PAGE_URL.
- Then create the 2nd scraping agent for details page and point it to first agent result using URL from source agent option in input.
See this complete documentation with an example Scraping Details Pages from Listings.
How to scrape websites with captcha?
Scraping CAPTCHA websites is not supported in self-services plans, but we provide Managed services of web scraping also known as Data As A Service with hybrid technology to bypass the CAPTCHA using human labor combined with bot for decoding CAPTCHA images with powerful OCR technology and manual typing when needed to continue the crawler to get the potential data you need for your business. Get in touch with Agenty sales team to request a quote.
This is how it works in backend :-)
How to modify a web scraping agent?
How do I download the agent result in CSV format?
How can I delete unused agents?
You may delete agents by visiting the agents page in your Agenty account > Then, select the agents by clicking on the check box and then hit the Delete button
How do I cancel my subscription?
If you've subscribed for Agenty plan and subscription is set to automatically renew on monthly basis you can cancel your subscription from your account page. To turn off the automatic renewal, login and then
- Go to your Setting > Subscription
- And click on the Cancel Subscription button as shown in screenshot.
Your account will remain active until your renewal date and after that you will no longer be charged and the account will be suspended.
Please note: when you hit the Cancel Subscription button it doesn't process immediately, instead the request is submitted with billing team to cancel the subscription which might take up to 24 hours to reflect the change in your account and you will receive a confirmation email as soon the cancellation process is completed.
How do I delete my Agenty account?
Here are the steps to delete your Agenty account (everything) permanently from Agenty Systems.
- Go to Settings from left menu and then scroll down
- You would see the Delete account section, click on it
- A dialog box will open, click on Delete my account button to delete your account
Note: Deleting account is an irreversible operation. Any data synced to your account will be deleted and no longer be accessible. Please ensure that you've signed out of all Agenty instances before you delete your account.