I try not to get too enamored with any given technology. Technology is a fickle mistress and changes often. It is far better to try and remain objective and bring a pragmatic approach to technology. But I can’t help myself, I am smitten with Lambdas. I love them. They scale horizontally beautifully and effortlessly. More importantly, they turn themselves off and stop billing when idle. What more could you want?
However, for all the good they provide straight out of the box, there needs to be a shift in the developer’s thinking to fully utilize them. At some point, a Lambda is going to need a password or token that needs to be secure. In the old world order, using servers, it is not uncommon to have a hidden file that is deliberately placed in the .gitignore so it is never checked in. A developer working locally can modify the file with some placeholder values and then, post deployment, a trusted person can ssh into the server and modify the file if need be.
This method of keeping secrets is a bit of a conundrum for Lambdas. They are designed to be stateless. If a developer were to check a password or token into git, even if it is a private repo, it greatly increases the number of people with access to this sensitive information. The less people that have database access to a production system, the better. Why advertise it by keeping a record of it in source control?
There are some ways to handle this situation. Lambdas allow environmental variables right from the console or deployed with a service such as serverless. The values here are available to the code:
Unfortunately, anyone with access to the AWS console can see these values in clear text. Given that I have code in a public repo running against my AWS account with elevated permissions capable of creating infrastructure and running all kinds of jobs on my dime, I care a lot about keeping it secure. Anyone who knows my secret can make a post to my Lambda and do all kinds of nefarious things. In the screenshot above, the variable for GIT_SECRET looks pretty random because I encrypted the field.
My requirements for secrets were:
- All secrets had to be secure. Secure enough for me to keep them in a public repo without worrying about someone doing something bad with my code that would wind up costing me a lot of money.
- Secrets had to be stored in git. I like the idea of everything being consolidated in a single repo. If the secret changes, I want an audit trail of who changed it and when.
- I did not want to rely on AWS IAM permissions to make sure users could not find secrets stored on S3 or on the Lambda console itself.
Given my requirements, the solution is not too difficult. I created a KMS key that I used to encrypt my values. Once encrypted, I store them in git. When the Lambda spins up, it decrypts the secret and keeps it in memory. I was also careful to never log the secret which would defeat the whole purpose.
WARM LAMBDA vs COLD LAMBDA
While Lambdas are supposed to be stateless, it turns out they do retain state. I have seen my lambdas that I use for ReSTful interfaces take up to 2.5 seconds on an initial call. Subsequent calls then take less than 100 milliseconds. The reason? The Lambda is warm. While 2.5 seconds is an eternity in web time, the good news is the more the Lambda is used, the more likely it is to be warm and fast.
In my code, I create a variable that is global in scope and instantiated with a null value. When the Lambda has a task, it first takes a look at my secret variable. If the variable is null, it knows it needs to first set it to the decrypted value. If the variable is not null, the Lambda can keep on processing.
Lines 156 - 179 handle all of this logic here:
Lambdas offer so much in terms of flexibility, scalability, and cost control. They do, however, change the way certain things are done vs using dedicated servers. Security measures should never be sacrificed, but with a little thinking and creativity, developers can keep their secrets secure. While not necessarily intuitive, the benefits Lambdas bring make it well worth it.
No comments:
Post a Comment