Firestore Integration

Agenty Firestore integration allows you to send your web scraping agents result to Google Cloud Firebase's Firestore NoSQL database by connecting to your project id, collection using service credentials.

Firestore is a real-time NoSQL database built on Google cloud infrastructure to scales automatically.

The Agenty Firestore integration allows you to import your agents result to selected Firestore project database collection automatically.

Service account

Agenty requires the service credential JSON file to connect with your Google cloud services. So, the first step to start using Firestore plugin in Agenty is to get your service credential file.

If you have the key file already with Firestore permission? You can skip this step and jump to configure plugin directly.

A service account represents a Google Cloud service identity, such as code running on Compute Engine VMs, Server-less etc.

  1. Login to your Google cloud account and go to IAM & Admin page
  2. Click on the Service Accounts in left menu to go to service account page Go go Google service acocunt
  3. Click on the Create service account button to create a new service account. Then enter the name and description of your choice. Create Google cloud service acccount
  4. Add a role to give full read/write access to Firestore Database Add a Firestore role
  5. Click on the continue button and find the Create key button on next page. Click on the Create key button to download a JSON file that contains the private key and other information for server-to-server connection. Create Firestore key
  6. Click on the Create button and it will download the key file in JSON format. Create service credential key

Configure Firestore Plugin

  1. Login to your Agenty account
  2. Go to Plugins page
  3. Click on the Add button against the Firestore plugin Attach Firestore plugin
  4. The Firestore plugin configuration page will open
  5. Select the agent where you want to attach this plugin to trigger on job completion. Select the agent for Firestore integration
  6. Then upload the credentials file and enter the project id, collection name in plugin configuration Firestore plugin configuration
  7. The primary key field is optional. This can be used to specify a primary key field from your agent to overwrite existing document (if any). If not specified, Firestore will auto-generate primary key to insert as new document
  8. Save the plugin configuration to attach it with the agent selected.

When a primary key field is specified, Agenty will use that field value and send a overwrite request to Firestore. If the document does not exist, it will be created. If the document does exist, its contents will be overwritten with the newly provided data

Execution

Go to the agent page where you've attached this Firestore plugin. For example, I attached it to a web scraping agent with name : Firestore plugin test

And, start that agent. (You may also start the web scraping task using our API)

Start web scraping agent with FireStore plugin

It will take few seconds to complete the job, depending on the number of URLs you are crawling in this agent. So, wait for the completion.

web scraping result to insert on Firestore

Logs and Preview

Click on the logs tab and scroll to last lines to see the plugin execution logs. You will see the message with Added or Updated document with id successfully.

DATE LEVEL MESSAGE
2019-10-27 09:56:01.4120 TRACE Firestore plugin started with timeout: 15 minutes
2019-10-27 09:56:09.9411 TRACE Added document with ID : dQw9rBUZky5imz8CQabQ successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : wNIhVclyetms8SeIOEie successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : 2fWdILu7Mx0tPBI3gfoI successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : UmZ3ImbYCL2qeDAoSYlm successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : q1ouQlLle5MXQmUKR221 successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : 98TephxtXjd3uMIm4Zi8 successfully

Go to your Firestore project page - https://console.firebase.google.com/ and select your project. Then go to database > collection to see the result sent by Agenty.

Firestore database preview