Python scraping API?

When using Python API, how to correctly parse request content into readable (etc) format? Iv'e tried using a bunch of different modules, wondering what is the official recommended way? at very least breaklines in between items...

Posted by Erick Guerra 8 months ago


@Erick DataScraping API accepts the input in JSON format and all your requests must have "content-type": "application/json" for POST methods. And the data you send with API CALL depends on particular endpoint. Here are two examples :-

Start a agent : POST  /v1/start/{agentid}

import http.client

conn = http.client.HTTPSConnection("api.datascraping.co")

payload = ""

headers = {
    'x-datascraping-api-id': "<API ID>",
    'x-datascraping-api-key': "<API Key>",
    'content-type': "application/json"
    }

conn.request("GET", "/v1/start/<AGENT ID>", payload, headers)

res = conn.getresponse()
data = res.read()

print(data)

 

Get output of a agent : GET /v1/output/{agentid}

import http.client

conn = http.client.HTTPSConnection("api.datascraping.co")

payload = ""

headers = {
    'x-datascraping-api-id': "<API ID>",
    'x-datascraping-api-key': "<API Key>",
    'content-type': "application/json"
    }

conn.request("GET", "/v1/output/<AGENT ID>?offset=0&limit=1000", payload, headers)

res = conn.getresponse()
data = res.read()

print(data)

Posted by anonymous 8 months ago

Topic Closed! This question is closed and don't accept posts now.

Close me