How to export scraped data to CSV or excel automatically


I have configured the fields in my scraping agent successfully executed the scraping agent in desktop app, and also scheduled it. All looks good.

But how to export the data which i was getting in desktop app to excel or CSV file automatically. if this is possible i am also looking to create one more schedule for exporting in excel file at regular intervals.

Please help

Posted by 1 years ago

You'd need to edit your agent (Right click > Edit) from agent explorer panel. (if your agent is not in tree, you might need to put that in "My documents/Data Scraping Studio/My Agents" folder in order to make your agent visible in tree)

Edit a scraping agent

Then go to output tab and select the option "Save to Local Drive".

  • Give name to your output file with file type extension (CSV, TSV, JSON and XML is supported)
  • Click on the browse button to select a path where you want to export your data file each time the agent is executed manually or via schedule.
  • You may also select include headings, or append mode depends on your requirment.

export scraped data to csv

Posted by 1 years ago

Thank you soo much .. for your quick reply.. it worked :) .

Can we have the new excel file every time the schedule runs instead of appending or over writing the same file.

Posted by 1 years ago

No, only append or overwrite mode is supported. It's advised to use append mode instead seprate file every time to manage better. Because if you are running your agent 50 times in a day. Wouldn't it be better to make a single file instead 50?

Or you may use webhook feature to post the data on your server or SQL every time the agent is executed to make it more meaningful and use in your current solution.

Posted by 1 years ago

Yes, posting to SQL sever will be appropriate in my requirement.

I will create a table with same fields in my DB on SQL server and import the scrapped data into it.

it will be very helpful, if you could give me more insight (documentation or videos) on posting the data to SQL (as the link which you have shared is only showing how to post the data to text file on server). i have checked in your website and you tube channel but couldnt find. 

Thanks for your help.

Posted by 1 years ago

Once you recieved data in JSON format, you can write a insert statement with foreach loop or bulkcopy easily. Please open a new question about "How to import scraped data in SQL Database" with details if you are not sure.

Posted by 1 years ago

Topic Closed! This question is closed and don't accept posts now.

Close me