Synergy

“All other things being equal, the simplest solution is the best.” - Me

I started my career in earnest with a consulting firm that was a part of an accounting firm in what was known collectively as the Big Six. As the consulting firm was still a part of an accounting firm, we were subject to some of the rules that governed the people in the accounting and audit side. One of these rules was that we needed to collect a certain amount of “Continuing Education” credits every year. To this day, the concept of continuing education remains important to me and I try to do at least one side project a year.

The side projects allow me to act as product owner, engineer, and architect. I get to explore technologies that I would never have touched otherwise. Through side work, I have gained some level of proficiency with React and Webpack, experimented with Mongo, and learned NodeJS. Some of my earlier work is still up on github and I cringe at the repos holding the code. My first side project did not have the .idea folder in the .gitignore, did not minify my React, did not follow the flux pattern, and there are plenty of other things about it that I can criticize.

On the other hand, it was the first thing I had ever done with React. I also integrated Server Sent Events so that the browser stayed current with what was on the server without the user needing to refresh. A couple of years ago, that was pretty cutting edge, so it is not a total waste.

I had originally hosted my web application on Amazon Web Services (AWS) taking advantage of the free tier. In two months, my daughter and I raised over $2,000 for the Juvenile Diabetes Research Fund by selling ice cream and collecting donations. I also had a nice site that I could use to walk through working examples of my code that was 100% my intellectual property. Although I was out of the ice cream business, I liked having it up and running as a demo of what I could pull off in my spare time.

A year went by and I realized I was spending $35 a month to host services I was not really using. I ported everything out of AWS and placed it on Digital Ocean where I had a modest Linux server for $5 a month that was more than capable of handling the miniscule traffic I expected on the site.

It was during this port I realized that the code I had placed in github had “drifted” from the code that was on my server. I was using AWS Elastic Beanstalk to deploy my code and instead of pushing to github and then doing a deploy, I skipped a step and just deployed. Since it was just me working on it, it was not the end of the world, but it was a really bad habit. I was working on another side project and vowed to be more diligent.

To help me ensure that what was in github represented what was live, I wrote a really small hook on my server in Digital Ocean. Its purpose was to receive a message from github, make sure that the message was authentic, and if it was legitimate - grab the latest copy of the code and restart itself. With a single controller and a five line shell script, I formed the basis of my devops philosophy. With this side project, I would write a unit test, get it to pass locally, commit the code, and voila it was released to my server.

A short while later, I was introduced to a, um different, devops philosophy. My company’s idea was to have dedicated Jenkins servers poll a git repository. From there, Jenkins would produce an artifact based on a script not checked in anywhere. The artifact would then get zipped and sit in Sonotype. Immediately after being placed in Sonotype, Rundeck would kick off a script sitting in a completely different repository. The script would grab the artifact that was just zipped and created and bring the artifact down, unzip it, and do whatever the deploy script required. In short, there were a ton of moving parts.

At times, Sonotype ran out of disk space jamming the whole deploy process. Every day at 4 o’clock, Jenkins would get flooded with jobs and it would delay deployments by fifteen to twenty minutes. Occasionally, a new package would need to be installed on the Jenkins server requiring an act of both Congress and God. There were at least fifteen dedicated Jenkins servers - all with different libraries, at different versions, and either over capacity causing delays or under utilized wasting money.

The deployment process spread what I considered a single unit of work into multiple repositories. All of it was wildly undocumented, but pushing a change could involve touching 3-4 repos. I kept on looking at my five lines of bash and wanting to go back to that.

In AWS land, there are multiple tutorials on how to create a Lambda or a Data Pipeline. Most will walk the developer through a couple of clicks on the console and it might seem pretty quick and easy, but DON’T DO IT! Everything that is introduced to the AWS environment should be auditable, capable of being rolled back, and kept under source control. Keeping with my KISS philosophy, I wanted to take the ease of laptop ops, but force deployments through git. After thinking about it for the better part of a year, this is what I came up with:

When a developer commits code to github, an action hook is kicked off. A Lambda in my sandbox environment inspects the content of the hook. If the body of the payload matches the encryption code, the request is considered legitimate and github receives a 200. If it does not match, github receives a 401 unauthorized response. If the branch that got committed matches the name sandbox, an SNS Topic is fired off to the deployment lambda. If not, the branch name is compared to test or master. If it matches one of these, the request is forwarded to the appropriate endpoint in a different AWS environment. Otherwise, the branch is considered to be a feature branch and is not deployed.

The deploy Lambda pulls the code at the specified branch from git. It then looks for a deploy.sh script and attempts to run it. The stderr and stdout of the deploy script are captured in DynamoDb and also streamed to any browsers that are receiving updates for the repo. A developer who just pushed can “follow” the deployment process. The deploy Lambda comes equipped with:

• Amazon Linux
• 1.5 Gigs of RAM
• 512MB of disk space
• Java 8
• Node 6.1
• AWS-CLI
• Git
• Serverless 1.2

So long as the process can be completed in under five minutes, all is well. I built a “Build Board” to provide monitoring across all the environments. The Build Board was deployed using this process. Its deploy script simply does an npm install, webpack build, and uploads the contents of the build directory to an S3 bucket dedicated to static website hosting. The S3 bucket was defined as part of my ecosystem in a cloud formation template. All of my infrastructure, code, build instructions, and secrets are stored in 1 (or 2) repositories.

Here is a quick demo showing how I deployed a “hello world” lambda to three different environments all by using git. Github itself allows various controls that can help to restrict deployments for a true controlled and gated experience. The branches test and master can be protected branches forcing a pull request before allowing a merge. Contributors permissions may be set up so that a product owner’s approval would be required before the merge/deployment. While there are some limitations in this approach - namely that the build process must finish within five minutes and not exceed 512MB of disk space, this tool should work with most small services. It scales up to allow for lots of deployments during the day, but does not cost anything when sitting idle.

I plan on using this as I build out my own product. Your milage may vary, but all the code is under MIT License and freely available on github.