Secrets in a Serverless World

I try not to get too enamored with any given technology. Technology is a fickle mistress and changes often. It is far better to try and remain objective and bring a pragmatic approach to technology. But I can’t help myself, I am smitten with Lambdas. I love them. They scale horizontally beautifully and effortlessly. More importantly, they turn themselves off and stop billing when idle. What more could you want?

However, for all the good they provide straight out of the box, there needs to be a shift in the developer’s thinking to fully utilize them. At some point, a Lambda is going to need a password or token that needs to be secure. In the old world order, using servers, it is not uncommon to have a hidden file that is deliberately placed in the .gitignore so it is never checked in. A developer working locally can modify the file with some placeholder values and then, post deployment, a trusted person can ssh into the server and modify the file if need be.

This method of keeping secrets is a bit of a conundrum for Lambdas. They are designed to be stateless. If a developer were to check a password or token into git, even if it is a private repo, it greatly increases the number of people with access to this sensitive information. The less people that have database access to a production system, the better. Why advertise it by keeping a record of it in source control?

There are some ways to handle this situation. Lambdas allow environmental variables right from the console or deployed with a service such as serverless. The values here are available to the code:

Unfortunately, anyone with access to the AWS console can see these values in clear text. Given that I have code in a public repo running against my AWS account with elevated permissions capable of creating infrastructure and running all kinds of jobs on my dime, I care a lot about keeping it secure. Anyone who knows my secret can make a post to my Lambda and do all kinds of nefarious things. In the screenshot above, the variable for GIT_SECRET looks pretty random because I encrypted the field.

My requirements for secrets were:

1. All secrets had to be secure. Secure enough for me to keep them in a public repo without worrying about someone doing something bad with my code that would wind up costing me a lot of money.
2. Secrets had to be stored in git. I like the idea of everything being consolidated in a single repo. If the secret changes, I want an audit trail of who changed it and when.
3. I did not want to rely on AWS IAM permissions to make sure users could not find secrets stored on S3 or on the Lambda console itself.

Given my requirements, the solution is not too difficult. I created a KMS key that I used to encrypt my values. Once encrypted, I store them in git. When the Lambda spins up, it decrypts the secret and keeps it in memory. I was also careful to never log the secret which would defeat the whole purpose.

WARM LAMBDA vs COLD LAMBDA

While Lambdas are supposed to be stateless, it turns out they do retain state. I have seen my lambdas that I use for ReSTful interfaces take up to 2.5 seconds on an initial call. Subsequent calls then take less than 100 milliseconds. The reason? The Lambda is warm. While 2.5 seconds is an eternity in web time, the good news is the more the Lambda is used, the more likely it is to be warm and fast.

In my code, I create a variable that is global in scope and instantiated with a null value. When the Lambda has a task, it first takes a look at my secret variable. If the variable is null, it knows it needs to first set it to the decrypted value. If the variable is not null, the Lambda can keep on processing.

Lines 156 - 179 handle all of this logic here:

https://github.com/PokerGuy/synergy/blob/master/handler.js

Lambdas offer so much in terms of flexibility, scalability, and cost control. They do, however, change the way certain things are done vs using dedicated servers. Security measures should never be sacrificed, but with a little thinking and creativity, developers can keep their secrets secure. While not necessarily intuitive, the benefits Lambdas bring make it well worth it.

Synergy

“All other things being equal, the simplest solution is the best.” - Me

I started my career in earnest with a consulting firm that was a part of an accounting firm in what was known collectively as the Big Six. As the consulting firm was still a part of an accounting firm, we were subject to some of the rules that governed the people in the accounting and audit side. One of these rules was that we needed to collect a certain amount of “Continuing Education” credits every year. To this day, the concept of continuing education remains important to me and I try to do at least one side project a year.

The side projects allow me to act as product owner, engineer, and architect. I get to explore technologies that I would never have touched otherwise. Through side work, I have gained some level of proficiency with React and Webpack, experimented with Mongo, and learned NodeJS. Some of my earlier work is still up on github and I cringe at the repos holding the code. My first side project did not have the .idea folder in the .gitignore, did not minify my React, did not follow the flux pattern, and there are plenty of other things about it that I can criticize.

On the other hand, it was the first thing I had ever done with React. I also integrated Server Sent Events so that the browser stayed current with what was on the server without the user needing to refresh. A couple of years ago, that was pretty cutting edge, so it is not a total waste.

I had originally hosted my web application on Amazon Web Services (AWS) taking advantage of the free tier. In two months, my daughter and I raised over $2,000 for the Juvenile Diabetes Research Fund by selling ice cream and collecting donations. I also had a nice site that I could use to walk through working examples of my code that was 100% my intellectual property. Although I was out of the ice cream business, I liked having it up and running as a demo of what I could pull off in my spare time.

A year went by and I realized I was spending $35 a month to host services I was not really using. I ported everything out of AWS and placed it on Digital Ocean where I had a modest Linux server for $5 a month that was more than capable of handling the miniscule traffic I expected on the site.

It was during this port I realized that the code I had placed in github had “drifted” from the code that was on my server. I was using AWS Elastic Beanstalk to deploy my code and instead of pushing to github and then doing a deploy, I skipped a step and just deployed. Since it was just me working on it, it was not the end of the world, but it was a really bad habit. I was working on another side project and vowed to be more diligent.

To help me ensure that what was in github represented what was live, I wrote a really small hook on my server in Digital Ocean. Its purpose was to receive a message from github, make sure that the message was authentic, and if it was legitimate - grab the latest copy of the code and restart itself. With a single controller and a five line shell script, I formed the basis of my devops philosophy. With this side project, I would write a unit test, get it to pass locally, commit the code, and voila it was released to my server.

A short while later, I was introduced to a, um different, devops philosophy. My company’s idea was to have dedicated Jenkins servers poll a git repository. From there, Jenkins would produce an artifact based on a script not checked in anywhere. The artifact would then get zipped and sit in Sonotype. Immediately after being placed in Sonotype, Rundeck would kick off a script sitting in a completely different repository. The script would grab the artifact that was just zipped and created and bring the artifact down, unzip it, and do whatever the deploy script required. In short, there were a ton of moving parts.

At times, Sonotype ran out of disk space jamming the whole deploy process. Every day at 4 o’clock, Jenkins would get flooded with jobs and it would delay deployments by fifteen to twenty minutes. Occasionally, a new package would need to be installed on the Jenkins server requiring an act of both Congress and God. There were at least fifteen dedicated Jenkins servers - all with different libraries, at different versions, and either over capacity causing delays or under utilized wasting money.

The deployment process spread what I considered a single unit of work into multiple repositories. All of it was wildly undocumented, but pushing a change could involve touching 3-4 repos. I kept on looking at my five lines of bash and wanting to go back to that.

In AWS land, there are multiple tutorials on how to create a Lambda or a Data Pipeline. Most will walk the developer through a couple of clicks on the console and it might seem pretty quick and easy, but DON’T DO IT! Everything that is introduced to the AWS environment should be auditable, capable of being rolled back, and kept under source control. Keeping with my KISS philosophy, I wanted to take the ease of laptop ops, but force deployments through git. After thinking about it for the better part of a year, this is what I came up with:

When a developer commits code to github, an action hook is kicked off. A Lambda in my sandbox environment inspects the content of the hook. If the body of the payload matches the encryption code, the request is considered legitimate and github receives a 200. If it does not match, github receives a 401 unauthorized response. If the branch that got committed matches the name sandbox, an SNS Topic is fired off to the deployment lambda. If not, the branch name is compared to test or master. If it matches one of these, the request is forwarded to the appropriate endpoint in a different AWS environment. Otherwise, the branch is considered to be a feature branch and is not deployed.

The deploy Lambda pulls the code at the specified branch from git. It then looks for a deploy.sh script and attempts to run it. The stderr and stdout of the deploy script are captured in DynamoDb and also streamed to any browsers that are receiving updates for the repo. A developer who just pushed can “follow” the deployment process. The deploy Lambda comes equipped with:

• Amazon Linux
• 1.5 Gigs of RAM
• 512MB of disk space
• Java 8
• Node 6.1
• AWS-CLI
• Git
• Serverless 1.2

So long as the process can be completed in under five minutes, all is well. I built a “Build Board” to provide monitoring across all the environments. The Build Board was deployed using this process. Its deploy script simply does an npm install, webpack build, and uploads the contents of the build directory to an S3 bucket dedicated to static website hosting. The S3 bucket was defined as part of my ecosystem in a cloud formation template. All of my infrastructure, code, build instructions, and secrets are stored in 1 (or 2) repositories.

Here is a quick demo showing how I deployed a “hello world” lambda to three different environments all by using git. Github itself allows various controls that can help to restrict deployments for a true controlled and gated experience. The branches test and master can be protected branches forcing a pull request before allowing a merge. Contributors permissions may be set up so that a product owner’s approval would be required before the merge/deployment. While there are some limitations in this approach - namely that the build process must finish within five minutes and not exceed 512MB of disk space, this tool should work with most small services. It scales up to allow for lots of deployments during the day, but does not cost anything when sitting idle.

I plan on using this as I build out my own product. Your milage may vary, but all the code is under MIT License and freely available on github.

Austin's Dirty Little Secret... It Kinda Sucks

“Regrets? I’ve got a few,

But then again, too few to mention.” - Frank Sinatra, “My Way”

I had a guilty pleasure of reading a professional blow hard/recruiter’s blog. This wizard of the recruiting industry and judge of technical talent, besides having never worked in the tech business, only works with elite engineers and only has amazing opportunities. His yarns usually center around PhD’s in machine learning being highly sought after by local well-funded startups. Anyhoo, the author in one of his post’s bragged about telling someone to “pack up the family and move to Austin.” (Source) Without a job. The implication being that the risk he faced moving to Austin jobless would be more than made up for by the potential reward. Almost the modern day equivalent of, “There’s gold in them thare hills!”

I don’t believe in crying over spilled milk, but this was some serious crappy advice. I have worked professionally in Los Angeles, Portland, Seattle, Salt Lake City, Baltimore, Dallas, San Francisco, and Melbourne, Australia. Very recently, a former co-worker I have known for well over ten years asked me my opinion about moving from Seattle to Austin. I let him know, in no uncertain terms, that Austin really sucks. Here are some things that you will never see on this blow hard’s blog.

THE COST OF LIVING MYTH

In today’s connected world, it is super easy to get data. I have gone through various periods of absolute infatuation with the website/app Zillow. With it, I can quickly and easily see the house of real estate, see pictures, and get a 1-10 rating on the nearby schools. This app convinced me that living in California again was an absolute pipe dream. However, home prices on Zillow seem much lower than equivalent home prices in my adopted home of Seattle. Do not be deceived!

The property taxes will vary a bit from location to location throughout Settle, but I paid roughly 0.9% on my home in Redmond. It didn’t click with me when I was looking at homes that were 33% cheaper in Austin that my monthly payment would be almost exactly the same because my property taxes in Austin are an eye popping 3.1%. The data was all there, but I willfully ignored it until it was too late. While the overall price of a home may be significantly lower, the property taxes are so much higher that it becomes a wash.

When having this conversation amongst Austinites, they are quick to point out that Texas does not have a state income tax. Well, neither does the great State of Washington. The next question is, “Well, where do they get their money?” I can’t answer that question, but I do know the facts. No state income tax. A 1% higher sales tax with groceries and staples excluded. Much lower property taxes. When looking at real estate, please do not make my mistake and be sure to factor in these insane property taxes.

Yet, to make up for the equivalent real estate cost as my adopted home of Seattle, I have found the salaries to be 30-40% lower. Part of me always thought that if I didn’t find a fulltime role right away, I could always go back to contracting. In Seattle, the contract rates paid more than the fulltime rates, as they should. With contracting, there are inherent risks, no benefits, and no vacation. Yet in Austin, the contract rates pay far less than the already low, low salaries.

So, equivalent cost. Lower salaries. What more could you want?

AN ABBREVIATED LIST OF REAL THINGS THAT HAVE HAPPENED TO ME PERSONALLY OVER THE LAST FOUR YEARS…

Maybe it’s not fair to blame all of this on Austin, but this has never happened to me outside this market:

• Shown up to a formal interview. Been interviewed by multiple people as scheduled. One person on the interview loop does not show up and I am left in a room by myself for an hour. TWICE.
• Interviewed with a small consulting firm and was asked to spend the day doing work, for free, for the company as part of the process.
• Accepted an offer from a hiring manager only to have the hiring manager quit within six weeks. TWICE.
• Worked at a “startup” founded in 1997 that routinely claimed that they would be the next Facebook, but in the span of a year half the company either quit or got fired
• Worked at a startup that was in business for two years. The company consisted of twenty people and had a CEO, COO, CTO, Enterprise Vice President, two regular Vice Presidents, and two directors. 25% of the employees were direct relatives of the CEO. The company never made a dime in revenue. To this day, I am pretty sure the whole thing was a scam to bilk a rich, clueless investor, but this one company has now spawned two separate companies trying to peddle the same dumb idea as part of the broader Austin Software Ponzi ™ that runs rampant here. Rich clueless investors in fear of missing out give money hand over fist to non technical managers who promise to make them billions in tech. They hire their friends and family, produce nothing, then attempt to sell out. Sometimes they do, sometimes they don’t. If they don’t, they just leach onto the next clueless investor and the whole cycle repeats itself.

• Gone through three rounds of interviews. Received an offer for $30k less than my current salary which I was upfront with the internal recruiter on our first call. After telling her that the whole experience was extremely unprofessional and a waste of everyone’s time, she decided to call my current employer for a “salary verification” after I declined the offer.
• Stated every step of the way through the interview process that the only reason I was looking for a new job was I had a family health situation that was making it difficult for me to keep up my current high travel requirements. Multiple times, I asked about the travel requirements. Accepted the job, took a 40% pay cut and then was asked to travel just as much as my previous job.

OF COURSE, YOUR MILEAGE MAY VARY

Should no one move to Austin? No, of course not. But anyone doing so should do it with their eyes wide open. The cost of living here is really no better than larger cities such as Seattle that have more job opportunities and higher salaries. As I write this, I am sitting across the street from the Capital Factory, Austin’s largest tech incubator. Every single “CEO” in there truly believes they are going to have a unicorn ($1 billion) valuation. All believe they will follow the four steps from Eric Cartman’s meme to a tee. Most assuredly, it will not happen for the vast majority. While cronyism and nepotism is rampant just about everywhere, it is especially bad here in Austin.

If you are a young buck looking to make it in the tech scene and willing to job hop for a few years and gain experience, it might be well worth your time. If you are in mid career, coming here for an unknown could very well be career suicide. Take hede.