Day 14 - Pets vs. Cattle Prods: The Silence of the Lambdas

By: Corey Quinn (@quinnypig)
Edited By: Scott Murphy (@ovsage)

“Mary had a little Lambda
S3 its source of truth
And every time that Lambda ran
Her bill went through the roof.”

Lambda is Amazon’s implementation of a concept more broadly known as “Functions as a Service,” or occasionally “Serverless.” The premise behind these technologies is to abstract away all of the infrastructure-like bits around your code, leaving the code itself the only thing you have to worry about. You provide code, Amazon handles the rest. If you’re a sysadmin, you might well see this as the thin end of a wedge that’s coming for your job. Fortunately, we have time; Lambda’s a glimpse into the future of computing in some ways, but it’s still fairly limited.

Today, the constraints around Lambda are somewhat severe.

• You’re restricted to writing code in a relatively small selection of languages– there’s official support for Python, Node, .Net, Java, and (very soon) Go. However, you can shoehorn in shell scripts, PHP, Ruby, and others. More on this in a bit.
• Amazon has solved the Halting Problem handily– after a certain number of seconds (hard capped at 300) your function will terminate.
• Concurrency is tricky: it’s as easy to have one Lambda running as a time as it is one thousand. If they each connect to a database, it’s about to have a very bad day. (Lambda just introduced per-function concurrency, which smooths this somewhat.)
• Workflows around building and deploying Lambdas are left as an exercise for the reader. This is how Amazon tells developers to go screw themselves without seeming rude about it.
  
• At scale, the economics of Lambda are roughly 5x the cost of equivalent compute in EC2. That said, for jobs that only run intermittently, or are highly burstable, the economics are terrific. Lambdas are billed in Gigabyte-Seconds (of RAM).
• Compute and IO scale linearly with the amount of RAM allocated to a function. Exactly what level maps to what is unpublished, and may change without notice.
• Lambda functions run in containers. Those containers may be reused (“warm starts”) and be able to reuse things like database connections, or have to be spun up from scratch (“cold starts”). It’s a grand mystery, one your code will have to take into account.
• There are a finite list of things that can trigger Lambda functions. Fortunately, cron-style schedules are now one of them. The Lambda runs
• within an unprivileged user account inside of a container. The only place inside of this container where you can write data is /tmp, and it’s limited to 500mb.
• Your function must fit into a zip file that’s 50MB or smaller; decompressed, it must fit within 250MB– including dependencies.

Let’s focus on one particular Lambda use case: replacing the bane of sysadmin existence, cron jobs. Specifically, cron jobs that affect your environment beyond “the server they run on.” You still have to worry about server log rotation; sorry.

Picture being able to take your existing cron jobs, and no longer having to care about the system they run on. Think about jobs like “send out daily emails,” “perform maintenance on the databases,” “trigger a planned outage so you can look like a hero to your company,” etc.

If your cron job is written in one of the supported Lambda languages, great– you’re almost there. For the rest of us, we probably have a mashup of bash scripts. Rejoice, for hope is not lost! Simply wrap your terrible shell script (I’m making assumptions here– all of my shell scripts are objectively terrible) inside of a python or javascript caller that shells out to invoke your script. Bundle the calling function and the shell script together, and you’re there. As a bonus, if you’re used to running this inside of a cron job, you likely have already solved for the myriad shell environment variable issues that bash scripts can run into when they’re called by a non-interactive environment.

Set your Lambda trigger to be a “CloudWatch Event - Scheduled” event, and you’re there. It accepts the same cron syntax we all used to hate but have come to love in a technical form of Stockholm Syndrome.

This is of course a quick-and-dirty primer for getting up and running with Lambda in the shortest time possible– but it gives you a taste of what the system is capable of. More importantly, it gives you the chance to put “AWS Lambda” on your resume– and your resume should always be your most important project.

If you have previous experience with AWS Lambda and you’re anything like me, your first innocent foray into the console for AWS Lambda was filled with sadness, regret, confusion, and disbelief. It’s hard to wrap your head around what it is, how it works, and why you should care. It’s worth taking a look at if you’ve not used it– this type of offering and the design patterns that go along with it are likely to be with us for a while. Even if you’ve already taken a dive into Lambda, it’s worth taking a fresh look at– the interface was recently replaced, and the capabilities of this platform continue to grow.