How to combine 2 or more scraping agents result in one agent?

To combine multiple web scraping agents result with exact same fields, we can use the Agenty post-processing script feature to merge multiple data tables and then set the agent result back to Agenty.


// Get the current job output as DataTable
var table = Agenty.Cloud.GetAgentResult("first agent id");

// Get running job id
var jobId = table.ExtendedProperties["job_id"];

// Get other agents data
var table2 = Agenty.Cloud.GetAgentResult("second agent id");

// merge both tables
table.Merge(table2);

// set job id
table.ExtendedProperties["job_id"] = jobId;

// set the result back to table
table.SetAgentResult();

Remember to run this agent in last, as it will download the 1st agent result to combine. So, the first agent must be completed before the post-processing script is executed.

For example, I have 2 agents to scrape electronics and hardware category from an eCommerce website and want to combine the final result in a single CSV.

I can use this script to meet my need.

  • Go to the agent page
  • Click on the Scripting tab
  • Switch on the post-processing script
  • Enter the script with and change the agent id GetAgentResult("here") in get result function
  • Save and run