How can I create an API to scrape website data?

I want to access my scraping agent via API, so that I can send the request from my android app with parameters to scrape a website.

How can I extract the data using dynamically submitted URLs? and then start the scraping to extract the final result in JSON.

There are code sample available for Microsoft .Net to mange the web scraping API (See here), and also available on Github you can replicate in your program.

Here is sample in curl :

Send a HTTP PUT request to add urls in your scraping agent

curl -X PUT -H "Content-type: application/json" -d '{
    "type": "manual",
    "data": [
        "http://website.com/products",
        "http://website.com/products/page-1.html",
        "http://website.com/products/page-2.html"
    ],
    "reference": null
    }' 'https://api.agenty.com/v1/inputs/{AGENT ID}?apikey={API KEY}'

Then send a POST request to start the agent

curl -X POST -H "Content-type: application/json" -d '{
    "agent_id":"{AGENT ID}"
    }' 'https://api.agenty.com/v1/jobs/scraping/async?apikey={API KEY}'

And GET request finally to fetch the scraped result in JSON

curl -X GET -H "Content-type: application/json" 
   'https://api.agenty.com/v1/jobs/{JOB_ID}/result?apikey={API KEY}&offset=0&limit=1000'

Signup now to get 100 pages credit free

14 days free trial, no credit card required!