Shopify Integration : Scrape supplier website and upload product to Shopify store automatically

Agenty Shopify integration allow you to automatically crawl the products from your manufacturer or supplier websites(even password protected) and upload to your Shopify store, update prices, images and more.

Shopify is most popular eCommerce platform and used by 600,000+ businesses to create their online stores, without writing any code to make your website live.
All you need is — Signup and upload your products to create an eCommerce website in few minutes. And their built-in templates and apps marketplace makes it even more easier for businesses to launch their eCommerce store online and start selling products globally in no time.

Our Shopify integration allows you to :

  • Crawl manufacturer website to upload all products to your store
  • Schedule the crawler to find new products daily, weekly and upload to your store
  • Schedule the crawler to update price, out-of-stock, inventory and other fields automatically

With all the custom rule and logic’s before uploading to Shopify, for example:

  • Add a custom script to increase the price by 10% to add your margin.
  • Replace “USD” with “$” in price field or something else using the built-in post-processing functions
  • Remove “By manufacturer name” form product title
  • And any other custom rule…

Prerequisites

  1. Good knowledge of Agenty web scraping tool to scrape data from supplier websites
  2. Credentials of supplier website, if password protected. (see docs here to learn how to crawl password protected site)
  3. A Shopify store
  4. Agenty professional or higher plan to get access of Shopify plugin

Shopify API key

  • Go to your Shopify store
  • Then click on Apps menu in left navigation
  • Scroll down and you’ll find an option which says “Working with a developer on your shop? Manage private apps”

  • Click on the Manage private apps
  • Then click on the Create a new private app
  • Name it Agenty and enter your developer email

  • Give the permissions to read_products, write_products
  • Save to get the API keyand password for this app. This API key and password will be used in Agenty to authenticate and upload products.

Setup your scraping agent

The web scraping agents MUST have following fields with exact same name to map them with Shopify API for importing.

Field Name Data Type Description
url string (mandatory field) Product URL of vendor website, this will be used as reference to sync the products (update price, quantity etc.) and avoid duplicates
title string (mandatory field) Product title
vendor string Brand or vendor name
body string HTML body of the product
product_type string The type of products, e.g mobiles, books, laptops etc.
tags string Comma separated tags
images string Comma separated images URLs
price number Product price in decimal format. Agenty will automatically remove any non number or period character while converting the price into valid decimal format.
quantity number In stock quantity of products in number format. Agenty will automatically remove any non number character while converting the quantity into valid number format. By default : 1
variants string Comma separated names of variants (For example : Red, Black, Blue )
published boolean Add this field with true/false value to control if the product should be published or not. By default: true

If you do not know how to create a scraping agent? I’d recommend to watch this video here or read this post first.

Configure Shopify Plugin

  • Go to Agenty plugins page
  • Click on the Add button on Shopify plugin row

  • A new page will open where we need to select the web scraping agent, where the plugins will be attached and also enter our Shopify store URL, api key and password from step #1

  • Profit percentage : Use the profit percentage text box to add the margin percentage. Agenty will increase the extracted price by a given percentage number to add your profit margin automatically.

  • Rounds up : Use the rounds up switch to enable/disable the rounding off. When enabled, the price will be rounded by next/previous 10, for example 198.99 will become 200; and 192.5 will become 190.

  • Save the plugin by clicking on “Save” button.

Start the scraper

  • Go to your agent page
  • Click on the Start button. (Remember : Don’t enter all URLs in input, if you are testing the scraper because that will run longer and consume your pages
    credit while testing
    )

Alternatively, you can also use our API to start a scraping job

curl -X POST -H "Content-type: application/json" -d '{
    "agent_id":"{AGENT ID HERE}"
    }' 'https://api.agenty.com/v1/jobs/scraping/async?apikey={API KEY HERE}'

The job will take few minutes, depending on the number of web pages you are crawling and also your plan limitation.

Once the job has been completed, check the trace logs and you’ll find messages like :

Shopify plugin started with timeout: 15 minutes
Product created successfully- Shopify Id: 1644195479605
No updated available - Shopify Id: 1644196003893
Product created successfully- Shopify Id: 1644196003887
Product updated successfully- Shopify Id: 1644196003862

Check your store

Now the agent has been completed successfully, check your Shopify store and you’ll find that all the scraped products have been uploaded to your store.