Achieve business goals and gain the edge by migrating your architecture.

Paweł Majkutewicz

A long time ago in a galaxy far, far away there was application. A pretty standard one, some UI with a 

 database. After a while a few problems arose:

once a while app crashes, so someone must restart it

the same situation with backups, they are handled by a custom script

So decisions were made - the app needs to be updated:

all crashes must be handled without user interaction

Therefore in the next chapters, we will migrate existing data to new MongoDB cluster and we will handle AWS application "upgrade".

Migrating MongoDB instance to the cluster

Having one instance of MongoDB available for the app is fine (😉). It is cheap and fast to create and deploy. Manage it also doesn't require too much effort. But it comes with few drawbacks - when something goes wrong and your instance goes down you have to bring it back online. This forces you to think about backups/snapshots, configurations, connections and so one. And this is a thing that no one wants to do it.

Instead of that, we can have a cluster, that will have redundancy, replication, automatic snapshots, so a lot of tasks will be done automatically.

Dump data from the current instance of MongoDB

The first step is to fetch all existing data from the running instance. This will be done using 'mongodump' utility. Unfortunately, I didn't have MongoDB installed on my machine, so it was obvious for me to use docker instance - I'll be able to switch mongo version in no time.

After a quick research, I've come up with bash script for making a dump for me. Most of it is boilerplate code for error handling, but the main part goes like this:

$MONG_VER - mongo version, so we can easily switch between 3.6 and 4.x

$SRC_HOST:$SRC_PORT - points to running MongoDB instance

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance

--archive=/workdir/dump/dump.gz - tells 'mongodump' to create archive file in /workdir/dump/ so effectively file will be created in ./dump/ directory

 - guessing that those guys should have the best knowledge and experience with mongo databases. Having own cluster is not complicated, go to their site, register and create a new project. And in project dashboard hit Build a New Cluster button. Next select Cloud Provider & Region and Cluster tier.

The most important thing here is to choose the appropriate cluster tier for our needs. In this case, M2 would be fine, but it has one drawback - it does not support peering (described later on):

And that is why we went with M10 cluster. In terms of backup, Cloud Provider Snapshots are enough, in that case, so we choose it. Last options are Cluster name, and naming thing is the hardest thing so after a couple of hours (😉) I was able to hit Create Cluster and after a while cluster is up and ready.

To be able to restore/import data our data we have to allow access to DB. First, we have to create a user that we will use and we can do it using Security -> Database Access menu. We have there ADD NEW USER button. After clicking it we have to fill a new user form:

Also, we have to whitelist our IP (or IP from which restore will be done). We can do it using ADD IP ADDRESS button in Security -> Network Access. If restore is performed from our machine, then we can use ADD CURRENT IP ADDRESS button.

It is time to import dumped data to the newly created cluster. Same situation here got bash script with a lot of error handling. The most important line is:

$MONG_VER - mongo version, so we can easily to switch between 3.6 and 4.x,

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance,

--archive=/workdir/dump/dump.gz - our dump,

$DEST_USER, $DEST_PASS - user credentials created in mongo cluster,

$DEST_HOST - points to running mongo instance,

$SRC_DB_NAME and $DEST_DB - allow us to rename our database, eg. original name was my-app-prod and we want to have dev instance named my-app-dev.

$DEST_HOST domain name of the created cluster. You can get this after clicking CONNECT on cluster dashboard:

And copy host part from connection string:

And that is all folks. We have running mongo cluster with our database. Now it is time to migrate the application.

Migrating AWS app to high availability architecture

In this part, we will take care of our running app. And now instead of using UI to create/deploy we will write CloudFormation script. This will give us the possibility to quickly redeploy the app.

Let's start from the smallest parts and go up in the hierarchy.

Our application is containerized in docker, so our basic block is TaskDefinition. We define here which docker image to use and some container configuration. All parameters required by the app are exposed as Parameters.

Next, we wrap the app instance with service. Service that runs and maintains the requested number of tasks and associated load balancers. All traffic that goes to app goes through service listeners, so let's check them now, cause as you can see we have a dependency on two of it.

Basically listeners handle all incoming traffic, we have two listeners.

The first listener redirects all HTTP traffic to HTTPS port.

HTTPS traffic are redirected to application (target group).

Specifies a target group for an Application Load Balancer or Network Load Balancer. Also it does health checks of our app. This solves our problem, when app die then will be automagically restarted.

AWS::ElasticLoadBalancingV2::LoadBalancer

And now lets define out load balancer. Basically we are allowing traffic from ouw group and from chosen one in ParAllowedSecurityGroup parameter - this is UI security group.

AWS::AutoScaling::AutoScalingGroup and AWS::AutoScaling::LaunchConfiguration

This part allows us to control how much instances of our app are running. For now, we just need one instance, but when appropriate time and our app will have over 2000 million users like facebook have than we can just increase our instances count and handle traffic flawlessly (😉).

Of course we have few more minor things. For example, after deploying our application we are updating api domain therefore UI is instantaneously redirected to newly created cluster.

AWS::EC2::SecurityGroup and AWS::EC2::SecurityGroupIngress

We have also security groups configured. Group is only allowing traffic from own security group - to manage load balancing.

Migrating AWS app with MongoDB to HA architecture. Visit our blog to learn more!

Migrating application to high availability architecture

11 OCTOBER 2019 • 22 MIN READ