Being an eMobility leader in various projects across multiple startups, Piotr was responsible for OCPI and OCPP integration as well as EMP and CPO platforms development. With extensive expertise in the open standards field and eMobility tech solutions, he is leading teams of experienced developers and contributing with invaluable knowledge and experience.

Piotr Majcher

I was recently working on a project which involved modeling data stored in 

 DynamoDB. It was only a single table, so by ‘modeling’, I mean setting indexes that would support the required types of queries. I wanted to highlight the differences between effective querying using indexed tables and the process of simple scanning. In order to do that, I needed DynamoDB to be populated with a considerable amount of data. My initial thought was to write a script/application that populated DynamoDB using AWS SDK. However, there was also the option of preparing the file with data, sending it to S3, and synchronizing the DynamoDB table with it by using the Data Pipeline.

The second option sounds better, doesn’t it? Data Pipeline, of course, works in two directions – in our case, from and to DynamoDB. It may look like a simple backup mechanism, but it’s a very interesting method with plenty of potentials. As a result, I’ve decided to postpone writing about modeling and focus on an article that looks at Data Pipeline, so let’s get started.

We need a file which contains data in a format accepted by Data Pipeline. One of these is the popular CSV format, so let’s see how it looks:

You may be wondering why it doesn’t look like standard CSV data such as:

This kind of CSV assumes some sort of recognizable structure: each row has an id, user identifier, and so on. DynamoDB is a schemaless database and each row may contain a different set of data (apart from the primary key, which is mandatory). And this is why we need to assign a column name to each item. To sum it up, CSV supported by Data Pipeline contains rows separated by a new line and each row contains a single JSON with columns separated by commas. To demonstrate the Data Pipeline method, I have prepared a file containing 1 million rows (175MB) in about 7 seconds on my 4-year-old laptop. Now we need to send this file to S3, but this will take a little bit longer. So, let’s create a bucket with a single directory called initial-batch and upload the file to that directory; this gives us data in a location that is easily accessible to Data Pipeline. Now we’re ready to focus on the more interesting part - the Data Pipeline setup.

Let’s stop thinking in terms of AWS. Instead, let’s try to imagine a process that is going to take data from one location and save it to another. What kind of information would be needed as an input for such a process? The bases are simple, all we need are:

In terms of a general overview, that’s all you have: a start point and an endpoint. What, though, does it look like in AWS?

The actual process of creating a Data Pipeline is well described in official 

, but, to quickly summarise this, we need to create a pipeline configured to look for data in S3 (source) and send it to DynamoDB (destination). The pipeline, of course, maybe customized, so there is a wide range of parameters that may be involved to guarantee the desired result. Fortunately, most of them have a default value. I’m going to use only two of them which, in our case, are crucial:

Before we dig a little deeper into those two parameters, let’s see how our Data Pipeline looks:

Each module in the diagram has its own set of properties. There is a Configuration module containing information about a whole pipeline, like IAM roles, log locations and so on. The S3DataNode describes the source of the data. The EmrCluster represents a cluster used in migration. The EmrActivity describes the migration process. Finally, the DynamoDBDataNode represents the location where we want to save our data. Now we need to take a look at which modules are affected by the parameters previously allotted to this example:

 defines which part of provisioned DynamoDB throughput should be involved in the pipeline - it’s a parameter of DynamoDBDataNode. In production systems making backups, it’s crucial that nothing impacts on overall performance, so it has to be carefully calculated in terms of provisioned throughput and expected load. Our goal is to load data from S3 as quickly as possible, so let’s set this property to 1.

 is a property of Resources Node - in our case EmrCluster. All this does is define a timeout for the pipeline work.

These two parameters depend on each other indirectly. Low DynamoDB write throughput ratio may end up with a timeout error if Terminate after is too small. Moving on, let’s look at an example that uses both of these parameters.

I have uploaded a file containing one million rows of records to S3. Let’s assume we would like the pipeline to be finished within 20 minutes. It’s a simple equation with one unknown which is the number of Write Units saving rows in DynamoDB.

Number of Write Units = number of records * throughput ratio / expected duration in seconds = 1,000,000 * 1.0 / 1200 = 833

Therefore, we need 833 Write Units and two actions need to be taken:

set DynamoDB table to have 833 Write Capacity Units

set DynamoDB write throughput ratio, property of DynamoDBDataNode to 1.0

We need to remember that Terminate after is applied to the whole life of EC2 instances used in EmsCluster – don’t forget, startup time also counts. So, let’s add 10 minutes for startup and set Terminate after to 30 minutes. We are now ready to activate Data Pipeline.

After about half an hour we can start to gather some numbers. Let’s see some screens from the AWS console. The first one shows a summary of the Data Pipeline’s execution:

The whole pipeline took about 27 minutes. Let’s see how long it would take to migrate data from S3 to DynamoDB instance.

This chart shows the number of Provisioned and Consumed DynamoDB Write Units and demonstrates that the data migration took around 20 minutes as expected. All of the rows are in our DynamoDB instance:

Each service used in a Data Pipeline is charged separately. In my case it was S3, DynamoDB and EmrCluster. Apart from that we need to pay for Data Pipeline as well, let’s try to sum it up:

 It’s by far the cheapest service. I have uploaded a 170 MB file and, after that, Data Pipeline gets the file for further processing. What’s more it’s free even after the trial period because sending the file to S3 and transferring data inside a single AWS Region is free.

 This gives us 25GB of storage and 25 Read and Write Units for free ( and it’s not limited to just the first 12 months) so we only need to pay for provisioned Write Units. AWS has estimated the monthly price of using 833 Write Units to be $456.06. As I’ve reduced the number of Write Units after the migration, I only need to pay for the time the migration took. DynamoDB charges an hourly rate so the final cost is about $0.6.

 We pay a per-second rate for the cluster and underlying EC2 servers, so assuming that the migration process took 30 minutes and we used two m3.large instances, we need to pay about $0.43.

 Free Tier offers three activities and three preconditions (I didn’t use it, my activity was launched manually). Without Free Tier, we need to pay $1 a month for High Frequency Activity (executed more than once a day) and $0.6 for Low Frequency Activity (executed once a day or less). We also need to pay for Data Pipeline when it’s inactive, so it’s a good idea to delete activity if you no longer need it. The cost of Data Pipeline is about $0.5 (half month of testing High Frequency Activity).

So, all in all, the whole cost of the migration is $0.6 + $0.43 + $0.5 = $1.53.

Setting up Data Pipeline between S3 and DynamoDB is a very simple example of how you can use this service. There’s a good chance that it may be able to bring a valuable feature to your application, but even outside of production it’s a fantastic option for an automatic backup. Not only can it aid sending your data for further computing, but it can also initialise your database.

How can you bootstrap DynamoDb using Amazon Data Pipeline? Case study, step by step.

Learn more about the advantages of both solutions Amazon Web Services and Oracle Cloud 

After founding Solidstudio, Pawel was an originator in terms of pursuing expertise in eMobility. With the build-up of a solid knowledge-base on industry’s needs & wants, he now keeps a close eye on products’ development. He’s also hugely responsible for the business’ side and designs the business models for our products.

Paweł Małkowiak

During the last decade, cloud computing has been growing at an exponential rate. The size of the cloud computing market is today larger than ever. COVID-19 outbreak accelerated digital transformation, making cloud computing a priority of choice. In 2020, the public cloud services market reached $257.5 billion in 2020. And according to Gartner forecasts, in 2021 the market revenue will increase 18.4% to a total of $304.9 billion.

Naturally, the cloud computing market has witnessed the rise of several players who now dominate the scene. Gartner's 2020 Magic Quadrant for Public Cloud Infrastructure Professional and Managed Services, positions:

If your organization aims to become more responsive to market changes and increase its operational efficiency, moving to the cloud is a smart move. And sooner or later, you'll have to choose a cloud provider. That's why we are here to give you a helping hand.

Why compare Oracle Cloud and Amazon Web Services? We decided to compare these cloud computing solutions delivered by two different vendors – a market leader and a smaller, but promising provider – to show you their pros and cons. Present how they differ in detail and indicate which factors you need to take into account when choosing the best cloud solution for your business.

Before we jump into details, let’s clarify some significant abbreviations, to be sure we’re on the same page. While browsing tips and clues on computing services you may find IaaS, PaaS, SaaS, DaaS and serverless. Following Microsoft and techopedia definitions:

IaaS - Infrastructure as a service - infrastructure is hosted on the public or private cloud instead of in a traditional on-premises data center. The infrastructure is delivered to customers on demand while being fully managed by the service provider.

PaaS - Platform as a service - offers the same kit as IaaS, plus additional features - middleware, development tools, business intelligence (BI) services, database management systems, and more. PaaS is designed to support the complete web application lifecycle: building, testing, deploying, managing, and updating.

SaaS - Software as a service - a complete software solution that you purchase on a subscription plan from a cloud service provider.

DaaS - Desktop as a service - a cloud computing solution in which virtual desktop infrastructure is outsourced to a third-party provider. DaaS relies on the virtual desktop, which is a user-controlled session or dedicated machine that transforms on-demand cloud services for users and organizations around the world.

Serverless - allows developers to build applications faster by cutting out the need to manage infrastructure. For serverless applications, the cloud provider automatically provisions, scales and manages the infrastructure required to run the code.

Note: the term ‘serverless computing’ applies to the event when the tasks linked with infrastructure provisioning and management are invisible to the developer. Servers are still running the code.

Oracle Cloud is a relatively new solution on the market and can't be compared to AWS in terms of scale. While Oracle Cloud might not occupy top positions in global cloud service provider rankings, the solution has a lot to offer. In fact, Oracle has set its cores on becoming a cloud leader in the future.

To help you recognize what Oracle Cloud is all about, let's explore its key features and capabilities.

By combining IaaS, PaaS, SaaS and DaaS solutions, Oracle provides a broad range of cloud services and products such as:

Business analytics, including Business Intelligence (BI) solutions, Big Data, data visualization, autonomous analytics, and more.

Application development, including mobile, Internet of Things (IoT), blockchain, database, and more.

Cloud infrastructure setup and maintenance.

Cloud for customer services, Customer Relationship Management (CRM) systems.

Security solutions focused on safety and compliance.

Oracle Cloud offers many services and products as an integrated set, allowing companies to lower their spending by eliminating the need to purchase third-party solutions. This is just one of the many advantages of Oracle. Check a detailed look at the pros of Oracle Cloud:

Integrated technology stack – Oracle delivers apps, middleware, and database as an entire, well-integrated system. There's no need to create a patchwork of third-party services coming from different vendors and go through the trouble of adapting them to the solution. To make the most of Oracle Cloud, it's best to invest in the all-inclusive version.

Smooth customization – Oracle allows its clients to be flexible in setting up the cloud service to match their unique requirements.

Cutting-edge technologies – Oracle is an industry leader whose advanced technologies make its cloud platform fast and high-performing. Speed is definitely a factor that sets this solution apart from others.

Scalable architecture – Oracle offers a sophisticated infrastructure that is fully scalable, allowing organizations to expand their IT platforms as they evolve. That feature is important for large enterprises that are going through the process of digital transformation and may want to revamp their operations in one area after another.

Niche focus – when building its cloud offer, Oracle focused on creating solutions that would match the requirements of specific customer segments. Instead of trying to grow its platform in every possible way, Oracle focuses on improving these capabilities.

Great customer support – you can be sure that your issue will be addressed quickly and effectively.

No solution is perfect, and Oracle Cloud also has its cons. Here are the most important disadvantages of this cloud vendor you should be aware of:

Poor low-end offering – since Oracle is focused on the high end, the offering around the low end is limited.

Poor performance with solutions based on .NET.

A minimum product might not be enough – Oracle provides all the services and products out-of-the-box, but these are very minimal solutions that might not be sufficient for companies that need simple functionalities. Organizations in need of a comprehensive platform are even less likely to get what they need.

Amazon Web Services (AWS) is a longstanding leader in providing cloud servers and storage to organizations looking to offload their data center infrastructures. In the fourth quarter of 2020, AWS generated revenue of approximately $12.7 billion.

Here's why so many companies choose AWS as their cloud provider today.

AWS offers cloud storage that supports managed, hybrid, block, and low-cost archive storages (such as Elastic File System, Storage Gateway, Glacier, S3, EBS), as well as a wide range of data transport functionalities.

Support for various databases (Aurora, RDS, Redshift, DynamoDB). Note: Aurora, Redshift, and DynamoDB are AWS custom technologies.

Support for advanced technologies (the Internet of Things, augmented reality and virtual reality, machine learning)

Tools that enable monitoring of workloads, configuration, and more.

Tools for security and performance optimization.

Analytics and business productivity, Business Intelligence (BI).

A broad range of developer tools (AWS Code Build, CodePipeline, CodeDeploy).

Top security, identity, and compliance management.

AWS has an excellent track record of successful implementations, and its financial performance attracts even more organizations to its offer.

Here are the key advantages of AWS over competing solutions:

Wide range of capabilities – compared to other cloud providers, Amazon offers the largest number and variety of functionalities. Large organizations invest in AWS because it's a sure-fire way to take full advantage of everything cloud technologies have to offer.

Easy implementation, migration, and management – AWS is surrounded by a large ecosystem of consulting partners and independent vendors who help companies to implement AWS-pre-integrated SaaS solutions.

Security and compliance – companies migrating to AWS can be sure that the processes they perform in the cloud are fully secure and compliant.

Excellent support – Amazon's global reach is reflected in the technical support it provides to the customers of AWS. The extensive multilingual customer support with high availability makes the solution one of the most reliable options on the market. The list of supported languages is constantly growing too.

Data centers around the world – Amazon operates multiple small data centers across the entire globe. This enhances service localization and limits the risk of local incidents.

Continuous offer development – AWS is constantly evolving and enriching its platform with new offerings. The vendor also improves existing functionalities, especially their integration and management.

Pay-as-you-go fee model – AWS offers a pay-as-you-go fee model where customers get to control over how many services they pay for, adding new ones as they grow. Minimizing upfront costs is easy with AWS. Companies can lay out expenses as they grow, without investing in products or services they don't need.

Still, AWS has its limitations that might inspire some customers to choose a different provider. Here are the most relevant disadvantages of AWS you need to take into account when making your decision:

The service implementation is complex – many AWS services require a lot of expertise when it comes to their implementation and support. Moreover, companies that want to make the most of them need to have a firm grasp of service updates and innovations. However, you can be sure that managed services providers know how to implement AWS technologies properly and can seek their assistance if you need it.

High pricing – an AWS-based solution might seem to come at a reasonable price. But if you want to create a custom solution that meets the specific requirements, be prepared for larger expenses than expected.

AWS or Oracle – which one should you choose?

The payment model of AWS allows smooth adaptability to market changes and fluctuations, helping your business stay nimble and invest in cloud computing at its own pace. Oracle Cloud also enables users to leverage flexibility, depending on how often customers need to rely on its services. AWS has established a large market presence and operates multiple data centers across the world. Their experience in delivering cloud services and excellent multilingual support is a significant competitive advantage.

Oracle Cloud and Amazon Web Services (AWS) are the two notable players in the cloud market. Choosing between the two options might be tricky, but we hope this guide helps you determine which platform will suit your business needs best.

Running a small startup or an enterprise-sized organization? It doesn’t matter - your data and cloud computing needs will rapidly grow, so you need a solution that will accommodate easily. At Solidstudio we help to fit the right solutions. If you still have some concerns, feel free to drop your e-mail. Or fill the contact form and take advantage of our digital transformation experience. Check back soon!

develops a fleet of software products designed to facilitate EV charging network management. See more about our 

Learn more about how Amazon Web Services (AWS) compares to Oracle Cloud platform and what to choose to fully benefit the cloud.

Oracle Cloud vs. AWS - Which one should you choose for your business?

Use case of dealing with AWS Lambda database connections.

Being a co-founder of Solidstudio, Paweł has got to see all the development phases. He’s a skilled professional with years of experience, both in the software and in the eMobility worlds.

Paweł Głogowski

In this article, I would like to describe a problem we faced with our 

 product. The whole application is hosted on 

. We use the AWS API gateway to serve the REST API along with 

 functions that handle requests. Code of functions is written in in 

 system to store data. As a database engine, we use 

Recently, we decided to test our product and see how many requests we could handle without any visible performance flaws. It turned out that after some decent (but not too heavy) load, lambdas stopped working with an error that indicated that the maximum number of database connections was reached. We needed to find a proper solution since our planned load is clearly much larger than our solution could handle.

We started with some background to understand the problem we have faced better.

How the connection pool works with sequelize

In the beginning, we need to better understand how the connection pool works. In general, the opening connection is a resource-consuming operation. Each open connection requires about 10MB of RAM and takes a decent amount of time. So it's common practice to create a connection, leave it open, and put it into a connection pool that can be used later on by the application and - what is most important - can be reused. It can really save a lot of RAM and other server resources.

In sequelize, you can specify the configuration of your pool like this. Take a look at the most important configuration parts.

 - the minimum number of connection in the pool (default: 0)

 - the maximum number of connection in the pool (default: 5)

 - maximum time, that a connection can be idle before being released. (default: 10000 ms)

 - time interval, after which sequelize-pool will remove idle connections.(default 1000 ms)

So after the connection is removed from the pool, it can be safely closed by the database engine.

On Amazon RDS, each database instance has a maximum number of connections that can be opened at the same time. It can be configured as a database parameter. By default, the value is calculated based on the memory available on the machine where the database is hosted. The formula is

It may look like it's an artificial boundary, but as soon as the number of connections is closer to that number, the performance drops dramatically, so increasing the number of connections above this calculated value is not an option.

Database connection in Postgresql can be in one of the following states:

active - This is currently running queries.

idle - this is where you have opened a connection to the DB (most frameworks do this and maintain a pool of them), but nothing is happening.

idle in transactions - this is where your app has run a BEGIN, but it's now waiting somewhere in a transaction and not doing any work.

According to that number of available connections for RDS instance looks like this:

td2.micro - 66td2.small - 150td2.medium - 312td2.large - 648m3.xlarge - 1237

Now we have covered the whole theory so we can see how it works with AWS lambda.

Exhausting database connections from Postgresql

In monolith applications, we have a straightforward situation. One application manages the database connection, and we can easily configure the number of connections in the pool. In distributed systems, it can be more complicated. However, in 

 with a shared database, the number of working instances is rather small, so it's quite easy to set a proper number of connections in the pool of each microservice.

The situation becomes quite interesting when it comes to lambdas. Usually, we need to configure the connection pool in the code of lambdas. Lambdas are executed in separate containers, which autoscale once more and more requests need to be handled. So for each lambda container, a connection pool is created. With the wrong configuration, it can lead to exhaust database connections really fast.

Let's analyze the results of our tests. We use jMeter to generate the desired amount of traffic. We started with 10000 users that generated about 100000 requests in 10 minutes, so it's ca. 167 requests per second. In the test results, I will not show how much time each request took - we will focus on the number of open connections. We use the RDS td2.medium instance for the sake of tests.

We started with the default sequelize configuration that is:

So it opens 5 new database connections for each lambda. We end up exceeding the number of connections, even for quite a small amount of users. See the chart below:

You can see that the number of connections was growing steadily but at one moment where lambda containers autoscale and each created 5 new connections and suddenly we went over the allowed number of connections.

Configuration changes to fix exhausting issues

There is a big spike - it means that lambdas need additional containers. Each of them requires 5 new connections, and they were created at the same time. We decided to reduce the number of maximum opened connections to 1. We were wondering how it affects the performance of requests, but the performance was even better than with the previous configuration. The explanation of that is quite straightforward. Lambdas are executed in parallel, however for one container execution is sequential. So even if several users share the same container, they are handled one by one.

So our first action was to reduce the numbers of max connections to 1, so our configuration looked like this:

Then the connections chart looks like this. It's much better than the previous, and the test eventually passed, but it almost reached the max number of connections. We still needed to find some other improvements to reduce the number of open connections.

We took a closer look at the idle parameters that are set on our tests. Idle determines how long a connection can leave in the pool before releasing it from the pool. It's set to 10 seconds by default. For our usage, it's definitely too much, lambda timeout is 6 seconds in our case, and on average, each lambda is executed in ~300 ms. So having 10 seconds means that some connections were opened after lambda finished its job. Also, they remain open even when lambda does not handle any requests. We reduced the idle time to 1 second and ran a test suite.

And the connection chart looks like this:

This means that we have, at most, only 48 open connections. This is far below the max limit, which is 312 for that database instance. So we were able to handle 167 requests per second without any problem.

However, we were wondering if reducing idle time further would result in better performance, but it turns out that we cannot lower idle connections since once the connection is released from the pool, it isn't automatically removed by the database engine. If lambda execution time is lower than the idle time it may result in opening a new connection to replace that one removed from the pool. Let's see how applications behave when we lower the idle parameter to 100ms.

As you can see, we end up with database connection exhaustion. The number of connections was quite steady at the beginning but grew dramatically as more and more requests came in.

Eventually, we left idle for 1 second, and we can easily handle ca. 167 requests per second on t2.medium RDS instance with great performance.

As the number of requests grows and grows, we will eventually need to use a bigger database instance like t2.large. However, keep in mind that upgrading databases without proper configuration is not an option, and even a powerful instance will not handle a larger amount of users without changing the configuration. We have tested a wrong configuration on M3.xlarge machine, and we suffer from connection exhaustion issues even for default configuration as well.

At Solid Studio Software House, our goal was to handle over 1000 requests per second without a significant performance drop. Due to the configuration changes described above, we managed to handle them with an average response time ca. 300 ms and RDS t2.xlarge instance. Serverless architecture requires a different perspective, but with proper configuration, it can perform really well without high costs.

Use case of dealing with AWS Lambda database connections. Click to learn more!

Handling AWS Lambda database connections

Achieve business goals and gain the edge by migrating your architecture.

Paweł Majkutewicz

A long time ago in a galaxy far, far away there was application. A pretty standard one, some UI with a 

 database. After a while a few problems arose:

once a while app crashes, so someone must restart it

the same situation with backups, they are handled by a custom script

So decisions were made - the app needs to be updated:

all crashes must be handled without user interaction

Therefore in the next chapters, we will migrate existing data to new MongoDB cluster and we will handle AWS application "upgrade".

Migrating MongoDB instance to the cluster

Having one instance of MongoDB available for the app is fine (😉). It is cheap and fast to create and deploy. Manage it also doesn't require too much effort. But it comes with few drawbacks - when something goes wrong and your instance goes down you have to bring it back online. This forces you to think about backups/snapshots, configurations, connections and so one. And this is a thing that no one wants to do it.

Instead of that, we can have a cluster, that will have redundancy, replication, automatic snapshots, so a lot of tasks will be done automatically.

Dump data from the current instance of MongoDB

The first step is to fetch all existing data from the running instance. This will be done using 'mongodump' utility. Unfortunately, I didn't have MongoDB installed on my machine, so it was obvious for me to use docker instance - I'll be able to switch mongo version in no time.

After a quick research, I've come up with bash script for making a dump for me. Most of it is boilerplate code for error handling, but the main part goes like this:

$MONG_VER - mongo version, so we can easily switch between 3.6 and 4.x

$SRC_HOST:$SRC_PORT - points to running MongoDB instance

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance

--archive=/workdir/dump/dump.gz - tells 'mongodump' to create archive file in /workdir/dump/ so effectively file will be created in ./dump/ directory

 - guessing that those guys should have the best knowledge and experience with mongo databases. Having own cluster is not complicated, go to their site, register and create a new project. And in project dashboard hit Build a New Cluster button. Next select Cloud Provider & Region and Cluster tier.

The most important thing here is to choose the appropriate cluster tier for our needs. In this case, M2 would be fine, but it has one drawback - it does not support peering (described later on):

And that is why we went with M10 cluster. In terms of backup, Cloud Provider Snapshots are enough, in that case, so we choose it. Last options are Cluster name, and naming thing is the hardest thing so after a couple of hours (😉) I was able to hit Create Cluster and after a while cluster is up and ready.

To be able to restore/import data our data we have to allow access to DB. First, we have to create a user that we will use and we can do it using Security -> Database Access menu. We have there ADD NEW USER button. After clicking it we have to fill a new user form:

Also, we have to whitelist our IP (or IP from which restore will be done). We can do it using ADD IP ADDRESS button in Security -> Network Access. If restore is performed from our machine, then we can use ADD CURRENT IP ADDRESS button.

It is time to import dumped data to the newly created cluster. Same situation here got bash script with a lot of error handling. The most important line is:

$MONG_VER - mongo version, so we can easily to switch between 3.6 and 4.x,

-v $(pwd):/workdir - this mounts current dir as /workdir in docker instance,

--archive=/workdir/dump/dump.gz - our dump,

$DEST_USER, $DEST_PASS - user credentials created in mongo cluster,

$DEST_HOST - points to running mongo instance,

$SRC_DB_NAME and $DEST_DB - allow us to rename our database, eg. original name was my-app-prod and we want to have dev instance named my-app-dev.

$DEST_HOST domain name of the created cluster. You can get this after clicking CONNECT on cluster dashboard:

And copy host part from connection string:

And that is all folks. We have running mongo cluster with our database. Now it is time to migrate the application.

Migrating AWS app to high availability architecture

In this part, we will take care of our running app. And now instead of using UI to create/deploy we will write CloudFormation script. This will give us the possibility to quickly redeploy the app.

Let's start from the smallest parts and go up in the hierarchy.

Our application is containerized in docker, so our basic block is TaskDefinition. We define here which docker image to use and some container configuration. All parameters required by the app are exposed as Parameters.

Next, we wrap the app instance with service. Service that runs and maintains the requested number of tasks and associated load balancers. All traffic that goes to app goes through service listeners, so let's check them now, cause as you can see we have a dependency on two of it.

Basically listeners handle all incoming traffic, we have two listeners.

The first listener redirects all HTTP traffic to HTTPS port.

HTTPS traffic are redirected to application (target group).

Specifies a target group for an Application Load Balancer or Network Load Balancer. Also it does health checks of our app. This solves our problem, when app die then will be automagically restarted.

AWS::ElasticLoadBalancingV2::LoadBalancer

And now lets define out load balancer. Basically we are allowing traffic from ouw group and from chosen one in ParAllowedSecurityGroup parameter - this is UI security group.

AWS::AutoScaling::AutoScalingGroup and AWS::AutoScaling::LaunchConfiguration

This part allows us to control how much instances of our app are running. For now, we just need one instance, but when appropriate time and our app will have over 2000 million users like facebook have than we can just increase our instances count and handle traffic flawlessly (😉).

Of course we have few more minor things. For example, after deploying our application we are updating api domain therefore UI is instantaneously redirected to newly created cluster.

AWS::EC2::SecurityGroup and AWS::EC2::SecurityGroupIngress

We have also security groups configured. Group is only allowing traffic from own security group - to manage load balancing.

Migrating AWS app with MongoDB to HA architecture. Visit our blog to learn more!

Bootstrapping Amazon DynamoDB using Amazon Data Pipeline.

18 JANUARY 2018 • 12 MIN READ