Shopify Integration : Scrape supplier website and upload product to Shopify store automatically

Agenty Shopify integration allow you to automatically crawl the products from your manufacturer or supplier websites(even password protected) and upload to your Shopify store, update prices, images and more.

Shopify is most popular eCommerce platform and used by 600,000+ businesses to create their online stores, without writing any code to make your website live. All you need is — Signup and upload your products to create an eCommerce website in few minutes. And their built-in templates and apps marketplace makes it even more easier for businesses to launch their eCommerce store online and start selling products globally in no time.

Our Shopify integration allows you to :

  • Crawl manufacturer website to upload all products to your store
  • Schedule the crawler to find new products daily, weekly and upload to your store
  • Schedule the crawler to update price, out-of-stock, inventory and other fields automatically

With all the custom rule and logic’s before uploading to Shopify, for example:

  • Add a custom script to increase the price by 10% to add your margin.
  • Replace “USD” with “$” in price field or something else using the built-in post-processing functions
  • Remove “By manufacturer name” form product title
  • And any other custom rule…

Prerequisites

  1. Good knowledge of Agenty web scraping tool to scrape data from supplier websites
  2. Credentials of supplier website, if password protected. (see docs here to learn how to crawl password protected site)
  3. A Shopify store
  4. Agenty professional or higher plan to get access of Shopify plugin

Shopify API key

  • Go to your Shopify store
  • Then click on Apps menu in left navigation
  • Scroll down and you’ll find an option which says "Working with a developer on your shop? Manage private apps"

shopify private app for agenty

  • Click on the Manage private apps
  • Then click on the Create a new private app
  • Name it Agenty and enter your developer email

create private app on shopify

  • Give the permissions to read_products, write_products
  • Save to get the API keyand password for this app. This API key and password will be used in Agenty to authenticate and upload products.

Setup your scraping agent

The web scraping agents should have following fields with exact same name to map them to your Shopify store

  • title : Title or the name of the product
  • vendor : Brand or vendor name
  • body : HTML body of the product
  • product_type : The type of products, e.g mobiles, books, laptops etc.
  • tags : Comma separated tags
  • images: A valid http/https image URL or base64_images field with valid base64 product image
  • price : Product price in decimal format. Agenty will automatically remove any non number or period character while converting the price into valid decimal format.
  • published : Add this field with true/false value to control if the product should be published or not. By default: true

shopify web scraper

If you do not know how to create a scraping agent? I’d recommend to watch this video here or read this post first.

Configure Shopify Plugin

  • Go to Agenty plugins page
  • Click on the Add button on Shopify plugin row

shopify plugin on agenty

  • A new page will open where we need to select the web scraping agent, where the plugins will be attached and also enter our Shopify store URL, api key and password from step #1

agenty shopify plugin details

  • Save the plugin by clicking on save button.

Start the scraper

  • Go to your agent page
  • Click on the Start button. (Remember : Don’t enter all URLs in input, if you are testing the scraper because that will run longer and consume your pages credit while testing)

crawl shopify website

Alternatively, you can also use our API to start a scraping job

curl -X POST -H "Content-type: application/json" -d '{
    "agent_id":"{AGENT ID HERE}"
    }' 'https://api.agenty.com/v1/jobs/scraping/async?apikey={API KEY HERE}'

The job will take few minutes, depending on the number of web pages you are crawling and also your plan limitation.

Once the job has been completed, check the trace logs and you’ll find messages like :

Shopify plugin started with timeout: 15 minutes
Product imported successfully- Shopify Id: 1644195479605
Product imported successfully- Shopify Id: 1644196003893

shopify web scraper logs

Check your store

Now the agent has been completed successfully, check your Shopify store and you’ll find that all the scraped products have been uploaded to your store.

shopify product catalog import automatically