Google Cloud has the fastest build now - a.k.a. හූ හූ, AWS!

Building a deployable serverless code artifact is a key functionality of our SLAppForge Sigma cloud IDE - the first-ever, completely browser-based solution for serverless development. The faster the serverless build, the sooner you can get along with the deployment - and the sooner you get to see your serverless application up and running.

AWS was "slow" too - but then came Quick Build.

During early days, back when we supported only AWS, our mainstream build was driven by CodeBuild. This had several drawbacks; it usually took 10-20 seconds for the build to complete, and it was rather repetitive - cloning the repo and downloading dependencies each time. Plus, you only get 100 free build minutes per month, so it adds a bit of a cost - despite small - to ourselves, as well as to our users.

Then we noticed that we only need to modify the previous build artifact in order to get the new code rolling. So I wrote a "quick build"; basically a Lambda that downloads the last build artifact, updates the zipfile with the changed code files, and re-uploads it as the current artifact. This was accompanied by a "quick deploy" that directly updates the code of affected functions, thereby avoiding the overhead of a complete CloudFormation deployment.

Then our ex-Adroit wizard Chathura built a test environment, and things changed drastically. The test environment (basically a warm Lambda, replicating the content of the user's project) already had everything; all code files and dependencies, pre-loaded. Now "quick build" was just a matter of zipping everything up from within the test environment itself, and uploading it to S3; just one network call instead of two.

GCP build - still in stone age?

When we introduced GCP support, the build was again based on their Cloud Build, a.k.a. Container Builder service. Although GCP did offer 3600(!) free build minutes per month (120 each day; see what I'm talking, AWS?), theirs was generally slower than CodeBuild. So, for several months, Sigma's GCP support had the bad reputation of having the slowest build-deployment cycle.

But now, it is no longer the case.

Wait, what? It only needs code - no dependencies?

There's a very interesting characteristic of Cloud Functions:

When you deploy your function, Cloud Functions installs dependencies declared in the package.json file using the npm install command.

-Google Cloud Functions: Specifying dependencies in Node.js

This means, for deploying, you just have to upload a zipfile containing the sources and a dependencies file (package.json, requirements.txt and the like). No more npm install, or megabyte-sized bundle uploads.

But, the coolest part is...

... you can do it completely within the browser!

jszip FTW!

That awesome jszip package does it all for us, in just a couple lines:

let zip = new JSZip();

files.forEach(file => zip.file(file.name, file.code));

/*
a bit more complex, actually - e.g. for a nested file 'folder/file'
zip.folder(folder.name).file(file.name, file.code)
*/

let data = await zip.generateAsync({
 type: "string",
 streamFiles: true,
 compression: "DEFLATE"
});

We just zip up all code files in our project, plus the Node/npm package.json and/or Python/pip requirements.txt...

...and upload them to a Cloud Storage bucket:

let bucket = "your-bucket-name";
let key = "path/to/upload";

gapi.client.request({
 path: `/upload/storage/v1/b/${bucket}/o`,
 method: "POST",
 params: {
  uploadType: "media",
  name: key
 },
 headers: {
  "Content-Type": "application/zip",
  "Content-Encoding": "base64"
 },
 body: btoa(data)
})).then(res => {
 console.debug("GS upload successful", res);

 return {
  Bucket: res.result.bucket,
  Key: res.result.name
 };
});

Now we can add the Cloud Storage object path into our Deployment Manager template right away!

...
{
 "name": "goofunc",
 "type": "cloudfunctions.v1beta2.function",
 "properties": {
  "function": "goofunc",
  "sourceArchiveUrl": "gs://your-bucket-name/path/to/upload",
  "entryPoint": ...
 }
}

So, how fast is it - for real?

1. jszip runs in-memory and takes just a few millis - as expected.
2. If it's the first time after the IDE is loaded, the Google APIs JS client library takes a few seconds to load.
3. After that, it's a single Cloud Storage API call - to upload our teeny tiny zipfile into our designated Cloud Storage bucket sigma-slappforge-{your Cloud Platform project name}-build-artifacts!
4. If the bucket is not yet available, and the upload fails as a result, we have two more steps - create the bucket and then re-run the upload. This happens only once in a lifetime.

So for a routine serverless developer, skipping steps 2 and 4, the whole process takes around just one second - the faster your network, the faster it all is!

In comparison to AWS builds, where we want to first run a dependency sync and then a build (each of which is preceded by HTTP OPTIONS requests, thanks to CORS restrictions); this is lightning fast!

(And yeah, this is one of those places where the googleapis client library shines; high above aws-sdk.)

Enough reading - let's roll!

I am a Google Cloud fan by nature - perhaps because my "online" life started with Gmail, and my "cloud dev" life started with Google Apps Script and App Engine. So I'm certainly at bias here.

Still, when you really think about it, Google Cloud is way simpler far more organized than AWS. While this could be a disadvantage when it comes to advanced serverless apps - say, "how do I trigger my Cloud Function periodically?" - GCF is pretty simple, easy and fast. Very much so, when all you need is a serverless HTTP endpoint (webhook) or bucket/queue consumer up and running in a few minutes.

And, when you do that with Sigma IDE, that few minutes could even drop down to a matter of seconds - thanks to the brand new quick build!

So, why waste time reading this - when you can just go and do it right away?!