Microservices Deployment using Docker
In one of our customer assignments, for the past 5 years with the monolithic architecture based product, we have been using the same, unchanged run-time environment with rpm based deployment. It also means that no container based deployment was used by then.
Early this year, as we adapted to microservices architecture, (in my previous post, I have covered how did we functionally decompose a monolythic application into microservices) we made a decision to run each and every microservice inside its own Docker container. Now that we’ve been in production for several months.
As far as working with Docker, it was not difficult to integrate it once we migrated our existing shell scripts to Dockerfile and created images.
We have one base image containing the OS and the Linux libraries we use across all images, and then a bunch of images built off that one. The size difference is only what was added on top of the base image. The ability to include diffs and build on top of existing containers pay benefits as it lets us continually roll out new versions as well as add code libraries, binary packages, and more to address particular use cases and solution sets. Using Docker for image management allows us to update images without any fear of breaking other parts of system. In other words, we can update the python2.7 image without touching the python3.4 image
The application source code, environment configuration files, dockerfile etc.. are hosted in Bitbucket. Developers would be able to checkin/ pull the code with this VCS.
Continuous Integration: Whenever a developer pushes new code into the bitbucket, building it along with unit and integration tests on a non-developer machine automatically is done via Jenkins Job
Docker Push (Deployment): End of any development cycle, the code is bundled and the updated docker image is pushed to the docker hub.
3. Docker Hub:
The docker’s registry capabilities for the docker images is provided by docker hub.
The Chef server acts as a hub for our infrastructure configuration data – cookbooks, secrets etc..
5. EC2 Instances with Chef-Client:
The EC2 instances nothing but the cloud boxes for running our applications, acts as nodes where the chef-clients are installed. The chef-clients running on the boxes regularly converge against the configuration policies that are stored on our Chef server.
Chef-client listens to docker hub if there are any changes in the docker image that the box already has. Say for example, chef-client running in stage environment where the authentication service docker container is up, listens to docker hub for the image ‘auth-service’ with the tag ‘stage’ for every configured (say 5) minutes. If it detects a change, then it stops the running docker container (auth-service), pulls the updated tagged docker image and run it. This way the deployment is automated with chef in place.
Docker in Different Environments:
We have multiple environments like dev, test, stage and prod. And now, we are left with the below questions.
- How do we pass environment related configuration information?
The environment related configuration information say the database server URL, rabbitMQ server URL etc… are kept in configuration files in any application. One configuration file per environment. Say application-test.properties, application-dev.properties, application-stage.properties etc… Chef-clients are up and running in all the boxes of all environments which are responsible to run a docker container from the updated docker image. The Chef-Clients are instructed via cookbooks in such a way that while running the docker container, they pass which environment it has to run. Say for example, Chef-Client on test box, passes the environment as ‘test’ where as chef-client on stage box passes the environment as ‘stage’. This is helpful in picking up the environment configuration file (application-dev.properties or application-stage.properties) the application server which has to be started inside the docker container should be fed with.
- How can we ensure that the image that was assured in QA is necessarily the same image in production as the image changes, every time when we rebuild. So how do we take Containers to Production with Confidence?
We essentially do not rebuild per environment. It means we develop, test and the cycle continues till we get stable build as shown below. Once the stability is reached, the stable docker image is tagged with to ‘stage’ and then as ‘prod’. The chef clients running in the instances of particular environment pulls the docker image of the respective tag if there is a change in the docker private registry
Docker Images in different env.
Docker has a great future and we’re glad we made the decision to include it our stack. If you want a manager for your infrastructure, yes, docker provides lightweight virtualisation with almost zero overhead. Also, docker is a great image distribution model for server templates built with Configuration Managers like Chef, Puppet etc…
So, what is missing in Docker? – Docker is quite stable and has proven to be truly production ready. But, the ecosystem is yet to get matured
- Logging – There’s no easy way logging with Docker. Need to go with a third party tool like Splunk, Sumologic, Logstash etc…
- Container Monitoring – Along with other monitoring strategies like system health monitoring, database monitoring, container monitoring must also be done which is not available off the shelf from docker. Need to rely on setting up monitoring tools like New Relic, Zabbix, Data Dog etc…
- Dangling Images: Docker doesn’t have an automatic garbage collection system as of now. If you execute ‘docker images -a’, you could see images with ‘<none>:<none>’. If we do not clear them out they can cause disk space problems. So we have a cronjob to clear these dangling images
Data in container: As Docker file system is ephemeral and when the container is shutdown, then the data stored in file system say application logs are lost. To overcome this, we have mapped host filesystem with the container file system; so that even if the container is destroyed, the data is backed up.