Serverless CSV Uploader for API integrations

At Haptik, thus far, we’ve developed Conversational AI solutions for a number of enterprises across sectors. Most of the bots we have developed so far have custom use cases which require API integrations. There are some use cases where the bot response is completely dependent on the user input. 

Some of the use-cases are as mentioned below:

  1. Documents required for loan type asked by the user.
  2. List of all available franchisee location for a given pin code.
  3. Create a ticket in client CRM when user post grievance.
  4. Calculation of premium for the term or investment plans
  5. Fetch the test report status

There are more such examples, but for now, I’ll discuss the one use case where the user asks the bot for report status.

In the above diagram, I’ll walk you through where API integration happens: The moment when our message pipeline receives a message it goes through the following stages:

  1. Language Detection
  2. Domain Classification
  3. Intent Detection
  4. Name Entity Recognition
  5. Deep Learning Layer
  6. Business Logic
  7. Multi-lingual support for ML models
  8. Bot personality & tone adaptable to context
  9. Data preprocessing
  10. Returns Bot Response

As can be observed from the above diagram, Named Entity Recognition returns entities from the message. In pointer E in the diagram, we use these entities as params to hit the client API, parse the API response, and return the bot response in the required format (Carousel, Quick Replies, Action Button Elements).

Problem Statement

While working with many enterprise businesses we noticed that some enterprises don’t always have APIs for use cases in which data doesn’t change frequently. for eg: franchise location or required document for some loan type.

To solve this problem, we used to:

  1. Ask enterprises to give us CSV dump for the data which we used to upload in MySQL table.
  2. Query the table for the given parameters.
  3. Return the response in appropriate HSL(Haptik Specific Language, which we use internally to create UI elements, like a carousel, quick replies, action buttons, etc) format. 

But with the above approach, we faced the following problems:

  1. Repetitive process, for each enterprise. The data fields were different so we were maintaining different SQL tables for each client.
  2. Frequent code release required on addition/deletion of new data field: Since schema was required to change every time some new data field was added or removed, a new code release was required for that.
  3. Non-Scalable: considering the number of clients being added this was not a scalable way to handle the problem.

Solution

We moved to a more skeleton based approach and let the values for response etc. be dynamically generated from the bot builder.

We decided to leverage AWS lambda, S3, and MongoDB for this problem.

AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of the Amazon Web Services. It is a computing service that runs code in response to events and automatically manages the computing resources required by that code. 

  1. Configure an S3 bucket to trigger a lambda function every time when someone uploads a CSV file. 
  2. Triggered Lambda function which parses the CSV data fields creates the schema, validates the data for each field, uploads the CSV to and dump the data in mongo.
  3. We have also made use of Lambda Layers which helped us access more packages like Pandas etc.
  4. Render the error or success response when CSV upload gets completed or send an email accordingly. 

Sample code for the lambda function

Advantages:

  1. Lesser cost
  2. Code release only required when we need to change the MongoDB query. 
  3. Any person in the company who has access can use the tool and configure the validation, so less developer intervention required.
  4. Makes life easier for Bot builders.

Future Scope:

UI tool so anyone can configure the schema and required validation to upload the CSV.

Hope this blog helps you understand how we improved the process of building custom bot solutions for clients and used serverless Technology to achieve the same. We welcome your comments and feedback and would love to hear from you. We are building more solutions like this so follow us. We are hiring. Do visit our careers page.

Share
Written by: