How to extract prices from competitor websites?

Scraping product, prices and stock information from an eCommerce competitors websites is one of the real business use of web scraping. And most of the eCommerce companies rely on web scraping software's for price comparison, and uses this technique to compare their price to find out if they are offering the best price in the competition or not. And then strategic planning to increase margin-profit.

This entire process is called "competitive pricing analysis" or "pricing intelligence" in business. In this tutorial we will learn how to extract price from a simple product list website (category or search result page) from an eCommerce website.

Setup

To extract any site with Agenty's data scraping tool, we'd need to create a scraping agent for that website first. So in order to setup a scraping agent, I've already installed the chrome extension and then :

  1. Go to the the website page you are looking to extract products from and then enable the extension by clicking on the extension icon on right corner of browser.
  2. Click on the "New" button to add fields and "Asterisk (*)" button generate selectors for any HTML element you are looking to extract.

In this screenshot example below, I am adding these four fields mentioned below to extract from this products list page :

  • ProductName
  • ProductPrice
  • ProductImage
  • ProductCartLink

Field 1 - ProductName

The CSS selector worked for product_name is .list-group-item-heading which is displaying all 6 product names correctly in preview :

Prodcutname

[
 {
   "ProductName": "Product A"
 },
 {
   "ProductName": "Product B"
 },
 {
   "ProductName": "Product C"
 },
 {
   "ProductName": "Product D"
 }
]

Field 2 - ProductPrice

To extract the price, I found that .lead was the correct CSS selector which extracted all 6 products price available in this page along with their USD currency symbol, if we see the preview in screenshot below :

productprice

Field 3 - ProductImage

The CSS selector worked for ProductImage is .list-group-image and the attribute as src which is displaying the 6 images for all 6 products on this page correctly, as we can see in extracted result preview :

image

Field 4 - ProductCartLink

The CSS selector worked for ProductCartLink is .btn-success and the attribute as href which is displaying links for all 6 products correctly in extracted result preview :

cartlink

Result Preview

Once we are done with setting up all the fields, click on the "Preview" button to see the result preview :

[
  {
    "ProductName": "Product A",
    "ProductPrice": "$21.40",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/118"
  },
  {
    "ProductName": "Product B",
    "ProductPrice": "$16.80",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/119"
  },
  {
    "ProductName": "Product C",
    "ProductPrice": "$30.00",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/120"
  },
  {
    "ProductName": "Product D",
    "ProductPrice": "$16.25",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/121"
  },
  {
    "ProductName": "Product E",
    "ProductPrice": "$19.00",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/122"
  },
  {
    "ProductName": "Product F",
    "ProductPrice": "$23.50",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/123"
  }
]

Save & Execution

When you are satisfied with the extracted result preview click on the "Save" button to save this web scraper agent to Agenty cloud application where you can use the features like scheduling, batch url crawling and more advanced options to automate the screen scraping to extract prices from websites.

Note : You would need to enter your Email and Password to save the web scraper to Agenty app, instead your login credentials

login

Now login to Agenty and you'd see your agent in your "Scraping Agents" list or on your "Dashboard" where you can start the agent to scrape the data or schedule that to run when you want. In the screenshot below I ran the agent and it extracted the product and pricing details in table below.

Capture

Wants to try on your own?

  1. Sign in to Agenty and click on "New Agent" button
  2. Here you would find this same agent in "Sample agents" list  to get from

extract price agent demo

Try it out and let me know what you think in comments below.