How to pull source scraping agent fields to merge data automatically?

If you are scraping a website with multiple agents by connecting them with each other to scrape data like a workflow. For example - Scraping categories from a retail website and then, products of each category scraped by category scraper. Then you must be looking for a way to combine both agents data automatically. Here is how to do that -

We can simply use the Input type field option to pull any field from the source file. E.g. selected list in input or the source agent connected.

Add a input field

  • Edit your details agent
  • Go to fields and collections tab
  • Click on add field button and then select type: Input to pull any field from the source agent, as in screenshot below.
  • Type = Input
  • From = Field name in source agent

Rerun the agent

Now, you can re-run your web scraping agent and it will set the input type field value automatically as in the source agen.