Connecting to AWS DocumentDB from a Lambda function

Few weeks ago, AWS announced their own NoSQL, MongoDB-compatible database called DocumentDB.

I must say, as a fan of MongoDB, I was immediately pumped! 🤗

https://twitter.com/doitadrian/status/1083368723250139137

Okay, maybe that statement was a bit exaggerated 🤪. But yeah, I’ve always wanted to have a “natively supported” MongoDB in the AWS cloud. I was actually secretly hoping this would happen when I saw that MongoDB was a platinum sponsor of the AWS re:Invent 2018, but at the time that did not happen.

Well, now it’s here and my team and I immediately started playing with it, primarily to see how it performs in conjunction with Lambda functions and to determine if it would be a good fit for us here at Webiny.

After I’d created a basic DocumentDB cluster, the natural step was to try to connect to it. But, in order to do that, I soon realized there are a few things you need to know and take care of first.

So, in this post, I wanted to show you how to connect to a DocumentDB inside of your Lambda functions (using Node.js runtime) and also, as a small bonus (who doesn’t love bonuses 😄), how to do the same from your local machine.

Let’s dig in!

Before we start…

There are a couple of things you should consider before connecting to your DocumentDB cluster.

First of all, it is always deployed inside of a VPC. This means there isn’t a public endpoint via which you can connect from the outside world. For example, you cannot just connect to your cluster directly from your machine.

Amazon DocumentDB is virtual private cloud (VPC)-only and does not currently support public endpoints. Thus, you can’t connect directly to your Amazon DocumentDB cluster from outside of your VPC.

Amazon DocumentDB is virtual private cloud (VPC)-only and does not currently support public endpoints. Thus, you can’t connect directly to your Amazon DocumentDB cluster from outside of your VPC.

Also, upon connecting, except the usual username and password, you must also use a public key they provide in the form of a .pem file.

By default, a newly created Amazon DocumentDB cluster only accepts secure connections using Transport Layer Security (TLS). Before you can connect using TLS, you first need to download the public key for Amazon DocumentDB.

By default, a newly created Amazon DocumentDB cluster only accepts secure connections using Transport Layer Security (TLS). Before you can connect using TLS, you first need to download the public key for Amazon DocumentDB.

So make sure to download it before continuing. For more information on this topic, you can visit the official AWS and MongoDB docs.

Alright, now we have everything we need to know. Let’s see how to connect to DocumentDB from a Lambda function.

Establishing a connection from a Lambda function

Code-wise, the cool thing about DocumentDB is that you don’t have to install any additional JS libraries or SDKs. You use it like you would use a regular MongoDB, via the official NodeJS mongodb package.

So, setting up a connection shouldn’t be much of a problem if you’ve already done it before. You only have to make sure to pass the connection URI and the public key correctly into the MongoClient.connect method.

You can find the connection URI in the AWS DocumentDB Console:

Note that for our case, you only need “the base” of the connection URI in the “Connect to this cluster with an application” section. Basically, you just leave out the extra query parameters (since the Node.js mongodb driver will take care of the rest), like so:

mongodb://some_user:some_password@webiny-cloud.cluster-some-id.us-east-2.docdb.amazonaws.com:27017

And of course, don’t forget to replace <insertYourPassword> with your actual password.

Alright, now, let’s see how to use the connection URI and downloaded certificate in our code:

import { MongoClient } from "mongodb";
import caBundle from "raw-loader!./path/to/rds-combined-ca-bundle.pem";

let cachedDb = null;

function connectToDatabase() {
    if (cachedDb) {
        return Promise.resolve(cachedDb);
    }

    return MongoClient.connect(
        process.env.MONGODB_URI,
        { ssl: true, sslCA: caBundle }
    ).then(db => {
        cachedDb = db;
        return cachedDb;
    });
}

export const handler = async (event, context) => {
    const db = await connectToDatabase();
    // Do something with it...
};

Please note that, in my development flow, my code is compiled with Babel so there are things like async, await, and import. I’m also using Webpack, which enables me to use raw-loader. With it, I can conveniently import the .pem file the same way I would any other .js file. But, if you don’t use Webpack in your development flow, you can just copy/paste the whole content of the downloaded .pem file into a single .js file, like so:

export default `-----BEGIN CERTIFICATE-----MIID9DCCAtygAwIBAgIBQjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMxEzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVB...`;

Also, make sure to use dotenv if you’re not already, since using the environment variables is the most reliable and secure method for storing sensitive configuration values.

Alright, we are ready to test our Lambda function inside the AWS!

I assume you already got the “deployment” part figured out. If not, serverless framework is certainly a great way to do it.

If we try to run our deployed function via the Lambda Management Console, you will notice that it times-out!

Well, the DocumentDB connection cannot be established because, as already mentioned, our Lambda function and DocumentDB cluster must reside in the same VPC. To do that, just scroll down to the Network section and select the correct VPC, its subnets, and an adequate security group.

In my case, I had everything set up in my default VPC:

Now, if you try to run the same function again, the connection should be established successfully!

Connection management

In a serverless world, connection management is unfortunately still something that might give you headaches.

If you’ve noticed, in our Lambda function, we actually never ended the established DocumentDB connection, because we wanted to re-use it on every following call. But, after some time, this might cause problems because a number of active connections will rise, while the old connections are still alive (and not doing anything), and eventually you hit the limit of active connections. At that point, new connection attempts will be rejected, which is certainly undesirable.

You can check the current state of connections by executing the following command inside of your DocumentDB cluster:

db.serverStatus().connections

// Outputs the following:
{
  "current" : 16,
  "available" : 1143,
  "totalCreated" : 511
}

The same problem can also be found in the RDS world, where there are some solutions to mitigate this, like e.g., the serverless-mysql lib that actually has a built-in connection manager which handles this for you. Additionally, Serverless Aurora also received the Data API, which abstracts connection management from us and let us interact with the database over an HTTP API.

At Webiny, we will closely monitor this for now. If required, I might even try to tackle this problem in my spare time, in the form of a separate NPM package.

We’ll see what time brings, but who knows; maybe we’ll see a serverless — Data API — powered DocumentDB in the near future? 😉

To finish this post, let’s see how we can establish a DocumentDB connection from a local machine.

Establish a connection from your local machine

If you need to directly connect to the DocumentDB cluster from a local machine, you will need to do it via a jumpbox / SSH tunneling.

It’s actually pretty simple to setup. You only need to create a simple EC2 instance (t2.micro will suffice) in the same VPC. Once it’s up and running, fire up your favorite MongoDB client; I like to use Studio 3t.

Make sure to correctly enter all of the necessary information and to attach the downloaded certificate file.

With everything entered correctly, you should be able to connect successfully to your DocumentDB cluster.

Final thoughts

Except the mentioned connection management issue, everything else seems to be working pretty well. I could write about our initial experiences, but I think I might save that for another post.

https://twitter.com/doitadrian/status/1085253232966987776

Here at Webiny, we are certainly excited about the newest AWS addition and we’ll be using it to power our serverless CMS, which you will be able to check out here in a couple of weeks from now. In the meantime, you can follow us via Twitter if you want to know more.

I hope this article has helped you to successfully overcome these initial steps in your Lambda / DocumentDB journey. If you have any questions or you feel I’ve maybe forgotten to mention something, feel free to let me know!

---

Thanks for reading! My name is Adrian and I work as a full stack developer at Webiny. In my spare time, I like to write about my experiences with some of the modern frontend and backend web development tools, hoping it might help other developers. If you have any questions, comments or just wanna say hi, feel free to reach out to me via Twitter.