TL;DR
The QuickBuild/QuickDeploy feature described here is pretty much obsoleted by the test framework (ingeniously hacked together by @CWidanage), that gives you a much more streamlined dev-test experience with much better response time!
In case you hadn't noticed, we have recently been chanting about a new Serverless IDE, the mighty SLAppForge Sigma.
With Sigma, developing a serverless app becomes as easy as drag-drop, code, and one-click-Deploy; no getting lost among overcomplicated dashboards, no eternal struggles with service entities and their permissions, no sailing through oceans of docs and tutorials - above all that, nothing to install (just a web browser - which you already have!).
So, how does Sigma do it all?
In case you already tried Sigma and dug a bit deeper than just deploying an app, you may have noticed that it uses AWS CodeBuild under the hood for the build phase. While CodeBuild gives us a fairly simple and convenient way of configuring and running builds, it has its own set of perks:
- CodeBuild takes a significant time to complete (sometimes close to a minute). This may not be a problem if you just deploy a few sample apps, but it can severely impair your productivity - especially when you begin developing your own solution, and need to reflect your code updates every time you make a change.
- The AWS Free Tier only includes 100 minutes of CodeBuild time per month. While this sounds like a generous amount, it can expire much faster than you think - especially when developing your own app, in your usual trial-and-error cycles ;) True, CodeBuild doesn't cost much either ($0.005 per minute of
build.general1.small), but why not go free while you can? :)
Options, people?
Lambda, on the other hand, has a rather impressive free quota of 1 million executions and 3.2 million seconds of execution time per month. Moreover, traffic between S3 and Lambda is free as far as we are concerned!
Oh, and S3 has a free quota of 20000 reads and 2000 writes per month - which, with some optimizations on the reads, is quite sufficient for what we are about to do.
2 + 2 = ...
So, guess what we are about to do?
Yup, we're going to update our Lambda source artifacts in S3, via Lambda itself, instead of CodeBuild!
Of course, replicating the full CodeBuild functionality via a lambda would need a fair deal of effort, but we can get away with a much simpler subset; read on!
The Big Picture
First, let's see what Sigma does when it builds a project:
- prepare the infra for the build, such as a role and an S3 bucket, skipping any that already exist
- create a CodeBuild project (or, if one already exists, update it to match the latest Sigma project spec)
- invoke the project, which will:
- download the Sigma project source from your GitHub repo,
- run an
npm installto populate its dependencies, - package everything into a zip file, and
- upload the zip artifact to the S3 bucket created above
- monitor the project progress, and retrieve the URL of the uploaded S3 file when done.
And usually every build has to be followed by a deployment; to update the lambdas of the project to point to the newly generated source archive; and that means a whole load of additional steps!
- create a CloudFormation stack (if one does not exist)
- create a changeset that contains the latest updates to be published
- execute the changeset, which will, at the least, have to:
- update each of the lambdas in the project to point to the new source zip file generated by the build, and
- in some cases, update the triggers associated with the modified lambdas as well
- monitor the stack progress until it gets through with the update.
All in all, well over 60-90 seconds of your precious time - all to accommodate perhaps just one line (or how about one word, or one letter?) of change!
Can we do better?
At first glance, we see quite a few redundancies and possible improvements:
- Cloning the whole project source from scratch is overkill, especially when only a few lines/files have changed.
- Every build will download and populate the NPM dependencies from scratch, consuming bandwidth, CPU cycles and build time.
- The whole zip file is now being prepared from scratch after each build.
- Since we're still in dev, running a costly CF update for every single code change doesn't make much sense.
But since CodeBuild invocations are stateless and CloudFormation's resource update logic is mostly out of our hands, we don't have the freedom to meddle with many of the above; other than simple improvements like enabling dependency caching.
Trimming down the fat
However, if we have a lambda, we have full control over how we can simplify the build!
If we think about 80% - or maybe even 90% - of the cases for running a build, we see that they merely involve changes to application logic (code); you don't add new dependencies, move your files around or change your repo URL all the time, but you sure as heck would go through an awful lot of code edits until your code starts behaving as you expect it to!
And what does this mean for our build?
80% - or even 90% - of the time, we can get away by updating just the modified files in the lambda source zip, and updating the lambda functions themselves to point to the updated file!
Behold, here comes QuickDeploy!
And that's exactly what we do, with the QuickBuild/QuickDeploy feature!
Lambda to the rescue!
QuickBuild uses a lambda (deployed in your own account, to eliminate the need for cross-account resource access) to:
- fetch the latest CodeBuild zip artifact from S3,
- patch the zip file to accommodate the latest code-level changes, and
- upload the updated file back to S3, overriding the original zip artifact
Once this is done, we can run a QuickDeploy which simply sends an UpdateFunctionCode Lambda API call to each of the affected lambda functions in your project, so that they can scoop up the latest and greatest of your serverless code!
And the whole thing does not take more than 15 seconds (give or take the network delays): a raw 4x improvement in your serverless dev workflow!
A sneak peek
First of all, we need a lambda that can modify an S3-hosted zip file based on a given set of input files. While it's easy to make with NodeJS, it's even easier with Python, and requires zero external dependencies as well:
Here we go... Pythonic!
import boto3
from zipfile import ZipFile, ZipInfo, ZIP_DEFLATED
s3_client = boto3.client('s3')
def handler(event, context):
src = event["src"]
if src.find("s3://") > -1:
src = src[5:]
bucket, key = src.split("/", 1)
src_name = "/tmp/" + key[(key.rfind("/") + 1):]
dst_name = src_name + "_modified"
s3_client.download_file(bucket, key, src_name)
zin = ZipFile(src_name, 'r')
diff = event["changes"]
zout = ZipFile(dst_name, 'w', ZIP_DEFLATED)
added = 0
modified = 0
# files that already exist in the archive
for info in zin.infolist():
name = info.filename
if (name in diff):
modified += 1
zout.writestr(info, diff.pop(name))
else:
zout.writestr(info, zin.read(info))
# files in the diff, that are not on the archive
# (i.e. newly added files)
for name in diff:
info = ZipInfo(name)
info.external_attr = 0755 << 16L
added += 1
zout.writestr(info, diff[name])
zout.close()
zin.close()
s3_client.upload_file(dst_name, bucket, key)
return {
'added': added,
'modified': modified
}We can directly invoke the lambda using the Invoke API, hence we don't need to define a trigger for the function; just a role with S3 full access permissions would do. (We use full access here because we would be reading from/writing to different buckets at different times.)
CloudFormation, you beauty.
From what I see, the coolest thing about this contraption is that you can stuff it all into a single CloudFormation template (remember the lambda command shell?) that can be deployed (and undeployed) in one go:
AWSTemplateFormatVersion: '2010-09-09'
Resources:
zipedit:
Type: AWS::Lambda::Function
Properties:
FunctionName: zipedit
Handler: index.handler
Runtime: python2.7
Code:
ZipFile: >
import boto3
from zipfile import ZipFile, ZipInfo, ZIP_DEFLATED
s3_client = boto3.client('s3')
def handler(event, context):
src = event["src"]
if src.find("s3://") > -1:
src = src[5:]
bucket, key = src.split("/", 1)
src_name = "/tmp/" + key[(key.rfind("/") + 1):]
dst_name = src_name + "_modified"
s3_client.download_file(bucket, key, src_name)
zin = ZipFile(src_name, 'r')
diff = event["changes"]
zout = ZipFile(dst_name, 'w', ZIP_DEFLATED)
added = 0
modified = 0
# files that already exist in the archive
for info in zin.infolist():
name = info.filename
if (name in diff):
modified += 1
zout.writestr(info, diff.pop(name))
else:
zout.writestr(info, zin.read(info))
# files in the diff, that are not on the archive
# (i.e. newly added files)
for name in diff:
info = ZipInfo(name)
info.external_attr = 0755 << 16L
added += 1
zout.writestr(info, diff[name])
zout.close()
zin.close()
s3_client.upload_file(dst_name, bucket, key)
return {
'added': added,
'modified': modified
}
Timeout: 60
MemorySize: 256
Role:
Fn::GetAtt:
- role
- Arn
role:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/AmazonS3FullAccess
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.comMoment of truth
Once the stack is ready, we can start submitting our QuickBuild requests to the lambda!
// assuming auth stuff is already done
let lambda = new AWS.Lambda({region: "us-east-1"});
// ...
lambda.invoke({
FunctionName: "zipedit",
Payload: JSON.stringify({
src: "s3://bucket/path/to/archive.zip",
changes: {
"path/to/file1/inside/archive": "new content of file1",
"path/to/file2/inside/archive": "new content of file2",
// ...
}
})
}, (err, data) => {
let result = JSON.parse(data.Payload);
let totalChanges = result.added + result.modified;
if (totalChanges === expected_no_of_files_from_changes_list) {
// all izz well!
} else {
// too bad, we missed a spot :(
}
});Once QuickBuild has completed updating the artifact, it's simply a matter of calling UpdateFunctionCode on the affected lambdas, with the S3 URL of the artifact:
lambda.updateFunctionCode({
FunctionName: "original_function_name",
S3Bucket: "bucket",
S3Key: "path/to/archive.zip"
})
.promise()
.then(() => { /* done! */ })
.catch(err => { /* something went wrong :( */ });(In our case the S3 URL remains unchanged (because our lambda simply overwrites the original file), but it still works because the Lambda service makes a copy of the code artifact when updating the target lambda.)
To speed up the QuickDeploy for multiple lambdas, we can even parallelize the UpdateFunctionCode calls:
Promise.all(
lambdaNames.map(name =>
lambda.updateFunctionCode({ /* params */ })
.promise()
.then(() => { /* done! */ }))
.then(() => { /* all good! */ })
.catch(err => { /* failures; handle them! */ });And that's how we gained an initial 4x improvement in our lambda deployment cycle, sometimes even faster than the native AWS Lambda console!
No comments:
Post a Comment