Modify and Cleaning up the Extracted Data using JavaScript

Some times it’s seen that the data we extract from webpages are not in our desired format or not well cleaned to use directly in the system and a post-processing is required to polish the data. Today, we are releasing the version 1.6 of Data Scraping Studio with powerful JavaScript engine will allows you to write JavaScript functions to transform and apply on extracted results automatically. 

By enabling you to create JavaScript functions that modify the extracted data and then use it to automate, Data Scraping Studio lets you make the process faster by eliminating the need for any fine-tuning of data.

How it Works

  • Right click on the scraped data grid > Click on modify data using JavaScript

    transform extracted result
  • The JavaScript Editor window will open with scraped Data as an object
  • Write your JavaScript function using the parsed Data object and return the result back to Data
  • Click on the Execute the script button(Play icon) to debug the code.
  • Click on the Save the script button(Save icon) to save the script to use for automation, the default directory for scripts is (My Documents/Data Scraping Studio/Scripts)

    Modify Scraped Data using JavaScript

Apply the JavaScript function on Scraped results Automatically

Once JavaScript function is tested and you are satisfy with the output preview, you can edit your scraping agent and add the JavaScript in MODIFY DATA tab and save it back to execute in parallel while scraping with instant transformed result in output.

The JavaScript code will be executed one time for each URL scraped. For e.g if a list page URL extracted 20 records, those 20 records will be sent to JS Engine to transform those 20 records using the JavaScript function selected in setup and so on for the next URLs.

Cleaning up Data Scraped from the Web

Now, execute the scraping agent again and you will see JS engine will transform the scraped result as per JavaScript function automatically

modify the result after extraction


Example 1 : Add a Custom Field

var scrapedData = data;
for (var i in scrapedData){
   scrapedData[i].NewField  = "Some Value";
data = scrapedData;

This example will add a new field with name NewField and value as Some Value.

Example 2 : Replace Dollar($) symbol with USD in PRICE field of scraped data

var scrapedData = data;
for (var i in scrapedData){
    scrapedData[i].PRICE  = scrapedData[i].PRICE.replace("$","USD");
data = scrapedData;

This example will replace the ($) with USD

Example 3 : Filter out scraped data based on one or many conditions

var scrapedData = data;
var filteredData = scrapedData.filter(function (el) {
    return el.COLOR == 'Red';
data = filteredData;

This example is created using Array.prototype.filter(Ref) and will filter out everything except Red color.

Example 4 : Add a calculated field

var scrapedData = data;
for (var i in scrapedData){
    var price = parseInt(scrapedData[i].PRICE.replace("$",""));
    scrapedData[i].PRICE_WITH_TAX = price + (price * 12/100);
data = scrapedData;

This example will add a PRICE_WITH_TAX field by adding 12% tax in the price

Related Q & A

Close me