Thursday, March 22, 2018

Serverless ZipChamp: Update your zip files in S3, (almost) in-place!

Simple Storage Service, more commonly known as S3, is the second most popular among AWS-offered cloud services. Originally meant to be a key-value store, it eventually transformed into one of the world's most popular file storage services. Its simple (just like the name!), intuitive, filesystem-like organization, combined with handy features like bucket hosting and CDN support via CloudFront, has made S3 an ideal choice for content hosting and delivery among organizations of all sizes.

AWS S3: simple storage

However, when it comes to using S3 as a filesystem, there are some really cool features that I miss; and I'm sure others do, too. A good example is extra support for archive-type entries (e.g. zip files); right now if you want to change a zip file on S3, you need to download it into a real filesystem (local, EC2, etc.), unpack (unzip) it, make the modification, repack (zip) it and upload it back to S3. Just think how cool it would have been possible to do this in-place in S3 itself, just like how archives can be modified on-the-fly in OSs like Ubuntu!

Of course, my dream is out-of-scope for S3, as it is simply supposed to provide key-value type storage. But it would be really cool if AWS guys could provide it as a feature, or if I could hack together a way to do it in-cloud (in-AWS) without having to sacrifice my local bandwidth or spin up an EC2 instance every time a modification is needed.

And the good news is: thanks to the recent advent of AWS Lambda, we can now hack together a lambda function to do just the thing for us!

AWS Lambda: serverless compute

If you're a total newbie, AWS Lambda allows you to host a snippet of logic (well, maybe even a fully-fledged application) that can be triggered by an external event (like an API call or a timer trigger). No need to run a virtual machine or even deploy a container to host your code; and the best part is, you only pay for the actual runtime - makes sense, because there is practically nothing running (meaning zero resource consumption) while your lambda is idle.

Ah, and I lied.

The best part is, with the AWS free tier, you get a quota of 1 million executions per month, for free; and 3.2 million seconds of execution time, also for free. At full capacity, this means that your lambda can run one million times each month, every invocation taking up to 3.2 seconds, before AWS starts charging for it; and even that, in tiny increments (just $0.00001667 per GB-second).

Wow, I can't think of a better place to host my own S3 zip file editor; can you? I won't be using it much (maybe a couple times a month), and whenever I need it, it will just come alive, do its job, and go back to sleep. Now, how cool is that?!

As if that wasn't enough, downloading and uploading content between S3 and lambda (in the same region) is free of charge; meaning that you can edit all the files you want, no matter how large (imagine a huge archive where you need to change only a teeny weeny 1 KB configuration file) without spending an extra cent!

Okay, let's roll!

Sigma: Think Serverless!

We'll be using Sigma, the brand new serverless IDE, in our dev path. There are some good reasons:

  • Firstly, Sigma is completely browser-based; nothing to install, except for a web browser - which you already have!
  • Secondly, Sigma completely takes care of your AWS stuff, including the management of AWS entities (lambdas, S3 buckets, IAM roles and whatnot) and their interconnections - once you give it the necessary AWS account (IAM) permissions, you don't have to open up a single AWS console ever again!
  • Last but not the least, Sigma automagically generates and maintains all bits and pieces of your project - including lambda trigger configurations, execution roles and permissions and related resources under a single CloudFormation stack (kind of like a deployment "definition"; somewhat like AWS SAM in case you're familiar, but much more flexible as Sigma allows integration with already existing AWS entities as well).

(In fact, behind the scenes, Sigma also leverages this "edit a zip file in S3" approach to drive its new QuickBuild feature; which will go public pretty soon!)

If the above sounded totally Greek to you, chill out! Lambda - or AWS, for that matter - may not look totally newbie-friendly, but I assure you that Sigma will make it easy - and way much fun - to get started!


TL,DR: if you're in a hurry, you can simply open the ready-made sample from my GitHub repo into Sigma and deploy it right away; just remember to edit the two S3 operations (by clicking the two tiny S3 icons in front of the s3.getObject() and s3.putObject() calls) to point to your own bucket, instead of mine - otherwise, your deployment will fail!


Okay, time to sign up for Sigma; here's the official guide, and we'll be putting out a video pretty soon as well!

When you are in, create a new project (with a nice name - how about zipchamp?).

your brand new Sigma project

Sigma will show you the editor right away, with boilerplate code for a default lambda function named, well, lambda.js. (Sigma currently supports NodeJS, and we will surely be introducing more languages in the near future!)

Sigma showing our brand-new project 'zipchamp'

For this attempt, we'll only focus on updating small textual files inside an S3 archive (don't worry; the archive itself could still be pretty big!), rather than binaries or large files. We will invoke the lambda with a payload representing a map of file changes: with keys representing the paths of files inside the archive, and values representing their content (to be added/modified, or to be removed if set to null).

{
    "path": "path/to/zip/file/within/bucket",
    "changes": {
        "path/to/new/file/1": "content for file 1",
        "path/to/existing/file/2": "new content of file 2",
        "path/to/to-be-deleted/file/3": null
    }
}

Sending such a map-type payload to a lambda would be quite easy if we use JSON over HTTP; luckily lambda provides direct integration with API Gateway, just the thing we would have been looking for.

Firstly, let's add the jszip dependency to our project, via the Add Dependency toolbar button, bestowing upon ourselves the power to modify zip files:

'Add Dependency' button

searching for 'jszip' dependency

Time to add a trigger to our function, so that it can be invoked externally (in our case, via an HTTP request coming through API Gateway). Click the API Gateway entry on the Resources pane on the left, drag it into the editor, and drop it right on to the function header (contaning the event parameter highlighted in red, with a red lightning symbol in front):

dragging API Gateway trigger

Sigma will open a pop-up, asking you for the configurations of the API Gateway trigger that you are about to define. Let's define a new API having API Name zipchamp, with a Resource Path /edit which accepts requests of type (Method) POST and routes them into our lambda. We also need to specify a Deployment Stage name, which could practically be anything (just an identifier for the currently active set of "versions" of the API components; we'll stick to Prod.

API Gateway pop-up configuration

Since we want to modify different files at different times, we would need to also include the path of the file in the payload. It would be easy if we can define our payload format at this point, to prevent possible omissions or confusions in the future.

    /* The request payload will take the following format:
    {
        "path": "path/to/zip/file/within/bucket",
        "changes": {
            "path/to/new/file/1": "content for file 1",
            "path/to/existing/file/2": "new content of file 2",
            "path/to/to-be-deleted/file/3": null
        }
    }
     */

Now, assuming a payload of the above format, we can start coding our magical zip-edit logic.

Planning out our mission:

  • fetch the file content from S3
  • open the content via JSZip
  • iterate over the entries (filenames with paths) in the changes field of our payload:
    • if the entry value is null, remove it from the archive
    • otherwise, add the entry value to the archive as a file (with name set to the entry key); which would be an update or an insertion depending on whether the file already existed in the archive (we could identify this difference as well, by slightly modifying the algorithm)
  • once the modifications are done, we could have uploaded the modified payload directly to S3, but the upload requires us to know the size of the upload in advance; unfortunately JSZip does not provide this yet. So we'd have to
    • save the modified zip file to the filesystem (/tmp), and
    • upload the file to S3, via a stream opened for the saved file, and specifying the Content-Length as the size of the saved file

Let's start by require-ing the JSZip dependency that we just added (along with fs, which we'll need real soon):

let AWS = require('aws-sdk');
let JSZip = require("jszip");
let fs = require("fs");

const s3 = new AWS.S3();

exports.handler = function (event, context, callback) {
    /* The request payload will take the following format:

And defining some variables to hold our processing state:

    }
     */

    let changes = event.changes;
    let modified = 0, removed = 0;

First, we need to retrieve the original file from S3, which requires an S3 operation. Drag an S3 entry into the editor, and configure it for a Get Object Operation. For the Bucket, you can either define a new bucket via the New Bucket tab (which would be created and managed by Sigma at deployment time, on your behalf - no need to go and create one in the S3 console!) or pick an already existing one from the Existing Bucket tab (handy if you already have a bucketful of archives to edit - or a default "artifacts" bucket where you host your archive artifacts).

configuring the 's3.getObject()' operation

Here I have:

  • used an existing Bucket, hosted-archives: the "stage" where I wish to perform all my zip-edit magic,
  • selected Get Object as the Operation, and
  • picked the path param from the original payload (event.path) as the S3 Object Key, @{event.path} (basically the path of the file inside the bucket, which we are interested in fetching); notice the @{} syntax in the pop-up field, which instructs Sigma to use the enclosed content as JS code rather than a constant string parameter (similar to ${} in JS).

If all is well, once you click the Inject button (well, it only gets enabled when all is well, so we're good there!), Sigma will inject an s3.getObject() code snippet - rich with some boilerplate stuff - right where you drag-dropped the S3 entity. As if that isn't enough, Sigma will also highlight the operation parameter block (first parameter of s3.getObject()), indicating that it has understood and is tracking that code snippet as part of its behind-the-scenes deployment magic! Pretty cool, right?

    s3.getObject({
        'Bucket': "hosted-archives",
        'Key': event.path
    }).promise()
        .then(data => {
            console.log(data);           // successful response
            /*
            data = {
                AcceptRanges: "bytes", 
                ContentLength: 3191, 
                ContentType: "image/jpeg", 
                ETag: "\\"6805f2cfc46c0f04559748bb039d69ae\\"", 
                LastModified: , 
                Metadata: {...}, 
                TagCount: 2, 
                VersionId: "null"
            }
            */
        })
        .catch(err => {
            console.log(err, err.stack); // an error occurred
        });

Sigma detects and highlights the 's3.getObject()' operation!

In the s3.getObject() callback, we can load the resulting data buffer as a zip file, via JSZip#loadAsync(buffer):

        .then(data => {
            let jszip = new JSZip();
            jszip.loadAsync(data.Body).then(zip => {

            });

Once the zip file is loaded, we can start iterating through our changes list:

  • If the value of the change entry is null, we remove the file (JSZip#remove(name)) from the archive;
  • Otherwise we push the content to the archive (JSZip#file(name, content)), which would correspond to either an insertion or modification depending on whether the corresponding entry (path) already exists in the archive.
            jszip.loadAsync(data.Body).then(zip => {
                Object.keys(changes).forEach(name => {
                    if (changes[name] !== null) {
                        zip.file(name, changes[name]);
                        modified++;
                    } else {
                        zip.remove(name);
                        removed++;
                    }
                });

We also track the modifications via two counters - one for additions/modifications and another for deletions.

Once the changes processing is complete, we are good to upload the magically transformed file back to S3.

  • As mentioned before, we need to first save the file to disk so that we can compute the updated size of the archive:
                    let tmpPath = `/tmp/${event.path}`
                    zip.generateNodeStream({ streamFiles: true })
                        .pipe(fs.createWriteStream(tmpPath))
                        .on('error', err => callback(err))
                        .on('finish', function () {
    
                        });
  • Time for the upload! Just like before, drag-drop a S3 entry into the 'finish' event of zip.generateNodeStream(), and configure it as follows:
    • Bucket: the same one that you picked earlier for s3.getObject(); I'll pick my hosted-archives bucket, and you can pick your own - just remember that, if you defined a new bucket earlier, you will have to pick it from the Existing Bucket list this time (because, from Sigma's point of view, the bucket is already defined; the entry will be prefixed with (New) to make your life easier).
    • Operation: Put Object
    • Object Key: @{event.path} (once again, note the @{} syntax; the Sigma way of writing ${})
    • Content of Object: @{fs.createReadStream(tmpPath)} (we reopen the saved file as a stream so the S3 client can read it)
    • Metadata: click the Add button (+ sign) and add a new entry pair: Content-Length = @{String(fs.statSync(tmpPath).size)} (reporting the on-disk size of the archive as the length of content to be uploaded)

configuring the 's3.putObject()' operation

                    .on('finish', function () {
                        s3.putObject({
                            "Body": fs.createReadStream(tmpPath),
                            "Bucket": "hosted-archives",
                            "Key": event.path,
                            "Metadata": {
                                "Content-Length": String(fs.statSync(tmpPath).size)
                            }
                        })
                            .promise()
                            .then(data => {

                            })
                            .catch(err => {

                            });
                    });
  • Lastly, let's add a successful callback invocation, to be fired when our S3 uploader has completed its job, indicating that our mission completed successfully:
                            .then(data => {
                                callback(null, {
                                    modified: modified,
                                    removed: removed
                                });
                            })
                            .catch(err => {
                                callback(err);
                            });

Notice the second parameter of the first callback, where we send a small "summary" of the changes done (file modifications and deletions). When we invoke the lambda via an HTTP request, we will receive this "summary" as a JSON response payload.

Throw in some log lines and error handling logic, until our lambda starts to look pretty neat:

let AWS = require('aws-sdk');
let JSZip = require("jszip");
let fs = require("fs");

const s3 = new AWS.S3();

exports.handler = function (event, context, callback) {
    /* The request payload will take the following format:
    {
        "path": "path/to/zip/file/within/bucket",
        "changes": {
            "path/to/new/file/1": "content for file 1",
            "path/to/existing/file/2": "new content of file 2",
            "path/to/deleted/file/3": null
        }
    }
     */

    let changes = event.changes;
    let modified = 0, removed = 0;

    console.log(`Fetching ${event.path}`);
    s3.getObject({
        'Bucket': "hosted-archives",
        'Key': event.path
    }).promise()
        .then(data => {
            let jszip = new JSZip();
            console.log(`Opening ${event.path}`);
            jszip.loadAsync(data.Body).then(zip => {
                console.log(`Opened ${event.path} as zip`);
                Object.keys(changes).forEach(name => {
                    if (changes[name] !== null) {
                        console.log(`Modify ${name}`);
                        zip.file(name, changes[name]);
                        modified++;
                    } else {
                        console.log(`Remove ${name}`);
                        zip.remove(name);
                        removed++;
                    }
                });

                let tmpPath = `/tmp/${event.path}`
                console.log(`Writing to temp file ${tmpPath}`);
                zip.generateNodeStream({ streamFiles: true })
                    .pipe(fs.createWriteStream(tmpPath))
                    .on('error', err => callback(err))
                    .on('finish', function () {
                        console.log(`Uploading to ${event.path}`);
                        s3.putObject({
                            "Body": fs.createReadStream(tmpPath),
                            "Bucket": "hosted-archives",
                            "Key": event.path,
                            "Metadata": {
                                "Content-Length": String(fs.statSync(tmpPath).size)
                            }
                        })
                            .promise()
                            .then(data => {
                                console.log(`Successfully uploaded ${event.path}`);
                                callback(null, {
                                    modified: modified,
                                    removed: removed
                                });
                            })
                            .catch(err => {
                                callback(err);
                            });
                    });
            })
                .catch(err => {
                    callback(err);
                });
        })
        .catch(err => {
            callback(err);
        });
}

The hard part is over!

(Maybe not that hard, since you hopefully enjoyed the cool drag-drop stuff; we sincerely hope you did!)

Now, simply click the Deploy Project button on the toolbar (or the menu item, if you like it that way) to set the wheels in motion.

Sigma will guide you through a few steps for getting your lambda up and running: committing your code to GitHub, building it, and displaying a summary of how it plans to deploy zipchamp to your AWS account via CloudFormation.

deployment summary

Once you hit Execute, Sigma will run the deployment gimmicks for you, and upon completion display the URL of the API endpoint - the trigger that we defined earlier, now live and waiting eagerly for your zip-edit requests!

deployment completed

Testing the whole thing is pretty easy: just fire up your favourite HTTP client (Postman, perhaps, sir?) And send a POST request to the deployment URL from above; with a payload conforming to our smartly-crafted zip-edit request format:

POST /Prod/edit HTTP/1.1
Host: yourapiurl.execute-api.us-east-1.amazonaws.com
Content-Length: 139

{
    "path": "my-dir/my-awesome-file.zip",
    "changes": {
        "conf/config.file": "property.one=value1\nproperty.two=value2"
    }
}

If you are a curl guy like me:

curl -XPOST --data '{
    "path": "my-dir/my-awesome-file.zip",
    "changes": {
        "conf/config.file": "property.one=value1\nproperty.two=value2"
    }
}' https://yourapiurl.execute-api.us-east-1.amazonaws.com/Prod/edit

If you would rather have zipchamp play with a test zip file than let it mess around with your premium content, you can use a simple zip structure like the following:

file.zip:
├── a/
│   ├── b = "bb"
│   └── d/
│       ├── e = "E"
│       └── f = "F"
└── c = ""

with a corresponding modification request:

{
    "path": "file.zip",
    "changes": {
        "b": "ba\nba\r\nblack\n\n\nsheep",
        "a/d/e": null,
        "a/c": "",
        "a/b": "",
        "a/d/f": "G"
    }
}

which, upon execution, would result in a modified zip file with the following structure:

file.zip:
├── a/
│   ├── b = ""
│   ├── c = ""
│   └── d/
│       └── f = "G"
├── b = "ba\nba\r\nblack\n\n\nsheep"
└── c = ""

Obviously you don't need to pull out your hair, composing the JSON request payload by hand; a simple piece of code, like the following Python snippet, will do just that on your behalf, once you feed it with the archive path (key) in S3, and the subpaths and content (local filesystem paths) of files that you need to update:

import json

payload = {
 "changes": {
  "file path 1 inside archive": open("file path 1 in local filesystem").read(),
  "file path 2 inside archive": open("file path 2 in local filesystem").read(),
  "file path inside archive, to be deleted": None
 },
 "path": "s3://temp-playground/cf-shell.zip"
}
print json.dumps(payload)

Nice! Now you can start using zipchamp to transform your long-awaited zip files, without having to download them anywhere, ever again!

Ah, and don't forget to spread the good word, trying out more cool stuff with Sigma and sharing it with your fellow devs!

Look, look! EI on LXC!

Containers have landed. And they are conquering the DevOps space. Fast.

containers: the future (https://blog.alexellis.io/content/images/2017/02/coding_stacks.jpg)

Enterprise Integration has been leading and driving the business success of organizations for ages.

enterprise integration: also the future

Now, thanks to AdroitLogic IPS, you can harness the best of both worlds - all the goodies of enterprise integration (EI), powered by the flexibility of Linux containers (LXC) - for your rapidly scaling, EI-hungry business and enterprise!

IPS for on-premise enterprise integration!

In case you had already looked at IPS, you might have noticed that it only offers a VirtualBox-based, single-instance evaluation distribution - not the most realistic way to try out a supposedly highly-available, highly-scalable integration platform.

But this time we have something "real" - something you can try out on your own K8s (or K8s-compatible; say OpenShift) cluster. It offers much better scalability - you can deploy our high-performance UltraESB-X instances across up to 20 physical nodes, in unlimited numbers (just ping us if you need more!) - and flexibility - the underlying K8s infra is totally under your control; for upgrades, patches, HA, DR and whatnot.

Kubernetes: the helmsman of container orchestration

OpenShift Container Platform (https://blog.openshift.com/wp-content/uploads/openshift_container_platform.png)

But the best part is that your UltraESB-X enterprise integrator instances would be running right inside your K8s network, meaning that you can easily and seamlessly integrate them with all your existing stuff: services (or microservices), queues, CI/CD pipelines, messaging systems, management and monitoring mechanisms, you name it.

UltraESB-X: enterprise integration - the easy way!

You can run the new IPS installer from any machine that has SSH access to the cluster - even from within the cluster itself. The process is fairly simple: just

If all goes well, the mission will go like this:

               ***************************************
                      Welcome to IPS Installer!
               ***************************************

Loading configurations...

++ MASTER=ubuntu@ip-1-2-3-4
++ NODES=(ubuntu@ip-5-6-7-8 ubuntu@ip-9-0-1-2)
++ SSH_ARGS='-i /path/to/aws/key.pem'
++ DB_IN_CLUSTER=true
++ DB_URL='jdbc:mysql://mysql.ips-system.svc.cluster.local:3306/ips?useSSL=false'
++ DB_USER=ipsuser
++ DB_PASS='7h1Zl$4v3RyI95e~CUr#*@m0R+'
++ DB_NODE=
++ ES_ENABLED=false
++ ES_IN_CLUSTER=true
++ ES_HOST=elasticsearch.ips-system.svc.cluster.local
++ ES_PORT=9300
++ ES_NODE=
++ DOCKER_REPO=adroitlogic
++ DOCKER_TAG=17.07.2-SNAPSHOT
++ set +x

Checking configurations...

NOTE: DB_NODE was not specified, defaulting to ip-5-6-7-8.
NOTE: ES_NODE was not specified, defaulting to ip-9-0-1-2.
Configurations checked. Looks good.

IPS will download required Docker images into your cluster
(~550 MB, or ~400 MB if you have disabled statistics).

Agree? yes


Starting IPS installation...

IPS needs to download the MySQL Java client library (mysql-connector-java-5.1.38-bin.jar)
in order to proceed with the installation.
Please type 'y' or 'yes' and press Enter if you agree.
If you are curious to know why we do it this way, check out
https://www.mysql.com/about/legal/licensing/oem/#3.

Agree? 

At this point you can either accept the proposal (obvious choice) or deny it (in which case the installer would fail).

Should you choose to accept it:

Agree? yes


Starting IPS installation...

Preparing ubuntu@ip-5-6-7-8...
Connection to ip-5-6-7-8 closed.
client.key.properties                                                                                                                                                            100%   53     0.1KB/s   00:00
license.conf.properties                                                                                                                                                          100%   78     0.1KB/s   00:00
license.key.properties                                                                                                                                                           100%   53     0.1KB/s   00:00
mysql-connector-java-5.1.38-bin.jar                                                                                                                                              100%  961KB 960.9KB/s   00:00
Successfully prepared ubuntu@ip-5-6-7-8

Preparing ubuntu@ip-9-0-1-2...
Connection to ip-9-0-1-2 closed.
client.key.properties                                                                                                                                                            100%   53     0.1KB/s   00:00
license.conf.properties                                                                                                                                                          100%   78     0.1KB/s   00:00
license.key.properties                                                                                                                                                           100%   53     0.1KB/s   00:00
mysql-connector-java-5.1.38-bin.jar                                                                                                                                              100%  961KB 960.9KB/s   00:00
Successfully prepared ubuntu@ip-9-0-1-2

configserver-rc.yaml                                                                                                                                                             100% 1960     1.9KB/s   00:00
configserver-svc.yaml                                                                                                                                                            100%  415     0.4KB/s   00:00
elasticsearch-rc.yaml                                                                                                                                                            100% 1729     1.7KB/s   00:00
elasticsearch-svc.yaml                                                                                                                                                           100%  418     0.4KB/s   00:00
ips-admin.yaml                                                                                                                                                                   100% 1093     1.1KB/s   00:00
ips-stats.yaml                                                                                                                                                                   100%  684     0.7KB/s   00:00
ipsweb-rc.yaml                                                                                                                                                                   100% 5023     4.9KB/s   00:00
ipsweb-svc.yaml                                                                                                                                                                  100%  399     0.4KB/s   00:00
mysql-rc.yaml                                                                                                                                                                    100% 1481     1.5KB/s   00:00
mysql-svc.yaml                                                                                                                                                                   100%  360     0.4KB/s   00:00
namespace "ips-system" created
namespace "ips" created
clusterrole "ips-node-stats" created
clusterrolebinding "ips-node-stats" created
clusterrole "ips-stats" created
clusterrolebinding "ips-stats" created
clusterrole "ips-admin" created
clusterrolebinding "ips-admin" created
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "configserver" created
service "configserver" created
replicationcontroller "ipsweb" created
service "ipsweb" created

IPS installation completed!
The IPS dashboard will be available at https://ip-5-6-7-8:30080 shortly.

You can always reach us at
    info@adroitlogic.com
or
    https://www.adroitlogic.com/contact/

Enjoy! :)

That's it! Time to fire up the dashboard and get on with it!

IPS: enterprise deployment on a single dashboard!

A few things to note, before you rush:

  • the hostnames/IP addresses used in config.sh should be the same as those being used as node names on the K8s side. Otherwise the IPS components may fail to recognize each other and the master. For now, an easy trick is to directly use the K8s node names for MASTER and NODES parameters (oh, and don't forget DB_NODE and ES_NODE!), and add host entries (maybe /etc/hosts on the installer machine) pointing those names to the visible IP addresses of the actual host machines; until we make things more flexible in the not-too-distant future.
  • Docker images for IPS management components will start getting downloaded on demand, as and when they are defined on the K8s side. Hence it may take some time for the system to stabilize (that is, before you can access the dashboard).
  • Similarly, the UltraESB-X Docker image will be downloaded on a worker node only when the first ESB instance gets scheduled on that node, meaning that you might observe slight delays during the first few ESB cluster deployments. If necessary, you can avoid this by manually running docker pull adroitlogic/ips-worker:17.07.2-SNAPSHOT on each worker node.

With the new distribution, we have also allowed you to customize the place where you store your IPS configurations (MySQL) and statistics (Elasticsearch): you can either set DB_IN_CLUSTER (or ES_IN_CLUSTER) to false and specify an external MySQL DB (or ES server) using DB_HOST, DB_PORT, DB_USER and DB_PASS (or ES_HOST and ES_PORT), or set it to true and specify a node name where MySQL (ES) should be deployed as an in-cluster pod, using DB_NODE (ES_NODE). Using an external MySQL or ES instance may be handy for cases where your cluster has limited resources (especially memory) and you want to maximally utilize them for running ESB instances rather than allocating them for infrastructure components.

customizable installation (https://d30y9cdsu7xlg0.cloudfront.net/png/128607-200.png)

Additionally, now you can also disable some non-essential features of IPS, such as statistics, at installation itself; just set ES_ENABLED to false, and IPS will skip the installation of an ES container and also stop the collection of ESB statistics at runtime. This can be really handy if you are running ESBs in a resource-constrained environment - disabling ES can bring down the per-ESB startup memory footprint from 700 MB right down to 250 MB! (We are already working on a leaner statistics collector based on the lean and sexy ES REST client - along with some other cool improvements - and once it is out, you will be able to run stats-enabled ESBs at under 150 MB memory.)

The new release is so new that we barely had the time to write the official docs for it - but all the existing docs, including the user guide and samples are all applicable to it, with some subtle differences:

  • The ESB Docker image (ips-worker) uses a different tag - a steaming hot 17.07.2-SNAPSHOT, instead of the default 17.07.
  • In samples, while you previously had to use the host-only address of the VM-based IPS node for accessing deployed services, now you can do it in a more "natural" way - by using the hostname of any node in the K8s cluster, just as you would do with any other K8s-based deployment.

For those of you who are starting from scratch, we have included a tiny guide to get you started with kubeadm - derived from the official K8s docs - that would kick-start you with a fully-managed K8s cluster within minutes, on your favourite environment (bare-metal or cloud). We also ship a TXT version inside the installer archive, in case you want to read it offline.

get started right away with kubeadm! (http://makeawebsite.org/wp-content/uploads/2014/12/quickstart-guide-icon.jpg)

And last but not the least, if you don't like what you see (although we're pretty sure that you will!), you can purge all IPS-related things from your cluster (:sad_face:) with another single command, teardown.sh:

++ MASTER=ubuntu@ip-1-2-3-4
++ NODES=(ubuntu@ip-5-6-7-8 ubuntu@ip-9-0-1-2)
++ SSH_ARGS='-i /path/to/aws/key.pem'
++ DB_IN_CLUSTER=true
++ DB_URL='jdbc:mysql://mysql.ips-system.svc.cluster.local:3306/ips?useSSL=false'
++ DB_USER=ipsuser
++ DB_PASS='7h1Zl$4v3RyI95e~CUr#*@m0R+'
++ DB_NODE=
++ ES_ENABLED=false
++ ES_IN_CLUSTER=true
++ ES_HOST=elasticsearch.ips-system.svc.cluster.local
++ ES_PORT=9300
++ ES_NODE=
++ DOCKER_REPO=adroitlogic
++ DOCKER_TAG=17.07.2-SNAPSHOT
++ set +x
Starting IPS tear-down...

Tearing down master...
namespace "ips-system" deleted
namespace "ips" deleted
clusterrole "ips-node-stats" deleted
clusterrole "ips-admin" deleted
clusterrole "ips-stats" deleted
clusterrolebinding "ips-node-stats" deleted
clusterrolebinding "ips-admin" deleted
clusterrolebinding "ips-stats" deleted
Successfully tore down master

Tearing down ubuntu@ip-5-6-7-8...
Connection to ip-5-6-7-8 closed.
Successfully tore down ubuntu@ip-5-6-7-8

Tearing down ubuntu@ip-9-0-1-2...
Connection to ip-9-0-1-2 closed.
Successfully tore down ubuntu@ip-9-0-1-2

IPS tear-down completed!

Enough talking, time to jump-start your integration - this time, on containers!

Friday, March 9, 2018

No more running around the block: Lambda-S3 thumbnailer, nailed by SLAppForge Sigma!

In case you hadn't noticed already, I have been recently babbling about the pitfalls I suffered when trying to get started with the official AWS lambda-S3 example. While the blame for most of those stupid mistakes is on my own laziness, over-esteem and lack of attention to detail, I personally felt that getting started with a leading serverless provider should not have been that hard.

banging head against the wall

And so did my team at SLAppForge. And they built Sigma to make it a reality.

Sigma logo

(Alert: the cat is out of the bag!)

Let's see what Sigma could do, to make your serverless life easy.

how Sigma works

Sigma already comes with a ready-made version of the S3 thumbnailing sample. Deploying it should take just a few minutes, as per the Readme, if you dare.

In this discussion, let's take a more hands-on approach: grabbing the code from the original thumbnailing sample, pasting it into Sigma, and deploying it into AWS—the exact same thing that got me running around the block, the last time I tried.

As you may know, Sigma manages much of the "behind the scenes" stuff regarding your app—including function permissions, trigger configurations and related resources—on your behalf. This relies on certain syntactic guidelines being followed in the code, which—luckily—are quite simple and ordinary. So all we have to do is to grab the original source, paste it into Sigma, and make some adjustments and drag-and-drop configuration stuff—and Sigma will understand and handle the rest.

If you haven't already, now is a great time to sign up for Sigma so that we could start inspiring you with the awesomeness of serverless. (Flattery aside, you do need a Sigma account in order to access the IDE.) Have a look at this small guide to get going.

Sigma: create an account

Once you're in, just copy the S3 thumbnail sample code from AWS docs and shove it down Sigma's throat.

S3 thumbnail code pasted into Sigma

The editor, which would have been rather plain and boring, would now start showing some specks of interesting stuff; especially on the left border of the editor area.

operation and trigger indicators on left border

The lightning sign at the top (against the function header with the highlighted event variable) indicates a trigger; an invocation (entry) point for the lambda function. While this is not a part of the function itself, it should nevertheless be properly configured, with the necessary source (S3 bucket), destination (lambda function) and permissions.

trigger indicator: red (unset)

Good thing is, with Sigma, you only need to indicate the source (S3 bucket) configuration; Sigma will take care of the rest.

At this moment the lightning sign is red, indicating that a trigger has not been configured. Simply drag a S3 entry from the left pane on to the above line (function header) to indicate to Sigma that this lambda should be triggered by an S3 event.

dragging S3 entry

As soon as you do the drag-and-drop, Sigma will ask you about the missing pieces of the puzzle: namely the S3 bucket which should be the trigger point for the lambda, and the nature of the operation that should trigger it; which, in our case, is the "object created" event for image files.

S3 trigger pop-up

When it comes to specifying the source bucket, Sigma offers you two options: you could either

  • select an existing bucket via the drop-down list (Existing Bucket tab), or
  • define a new bucket name via the New Bucket tab, so that Sigma would create it afresh as part of the project deployment.

Since the "image files" category involves several file types, we would need to define multiple triggers for our lambda, each corresponding to a different file type. (Unfortunately S3 triggers do not yet support patterns for file name prefixes/suffixes; if they did, we could have gotten away with a single trigger!) So let's first define a trigger for JPG files by selecting "object created" as the event and entering ".png" as the suffix, and drag, drop and configure another trigger with ".jpg" as the suffix—for, you guessed it, JPG files.

S3 trigger for PNG files

There's a small thing to remember when you select the bucket for the second trigger: even if you entered a new bucket name for the first trigger, you would have to select the same, already-defined bucket from the "Existing Bucket" tab for the second trigger, rather than providing the bucket name again as a "new" bucket. The reason is that Sigma keeps track of each newly-defined resource (since it has to create the bucket at deployment time) and, if you define a new bucket twice, Sigma would get "confused" and the deployment may not go as planned. To mitigate the ambiguity, we mark newly defined buckets as "(New)" when we display them under the existing buckets list (such as my-new-bucket (New) for a newly added my-new-bucket) - at least for now, until we find a better alternative; if you have a cool idea, feel free to chip in!.

selecting new S3 bucket from existing buckets list

Now both triggers are ready, and we can move on to operations.

S3 trigger list pop-up with both triggers configured

You may have already noticed two S3 icons on the editor's left pane, somewhat below the trigger indicator, right against the s3.getObject and s3.putObject calls. The parameter blocks of the two operations would also be highlighted. This indicates that Sigma has identified the API calls and can help you by automatically generating the necessary bells and whistles to get them working (such as execution permissions).

S3 operation highlighted

Click on the first icon (against s3.getObject) to open the operation edit pop-up. All we have to do here is to select the correct bucket name for the Bucket parameter (again, ensure that you select the "(New)"-prefixed bucket on the "existing" tab, rather than re-entering the bucket name on the "new" tab) and click Update.

S3 getObject operation pop-up

Similarly, with the second icon (s3.putObject), select a destination bucket. Because we haven't yet added or played around with a destination bucket definition, here you will be adding a fresh bucket definition to Sigma; hence you can either select an existing bucket or name a new bucket, just like in the case of the first trigger.

S3 putObject operation pop-up

Just one more step: adding the dependencies.

While Sigma offers you the cool feature of the ability to add third-party dependencies to your project, it does need to know the name and version of the dependency at build time. Since we copied and pasted an alien block of code into the editor, we should separately tell Sigma about the dependencies that are being used in the code, so that it can bundle them along with our project sources. Just click the "Add Dependency" button on the toolbar, search for the dependency and click "Add", and all the added dependencies (along with two defaults, aws-sdk and @slappforge/slappforge-sdk) will appear on the dependencies drop-down under the "Add Dependency" button.

Add Dependency button with dependencies drop-down

In our case, keeping with the original AWS sample guidelines, we have to add the async (for waterfall-style execution flow) and gm (for GraphicsMagick) dependencies.

adding async dependency

Done!

Now all that remains is to click the Deploy button on the IDE toolbar, to set the wheels in motion!

Firstly, Sigma will save (commit) the app source to your GitHub repo. So be sure to provide a nice commit message when Sigma asks you for one :) You can pick your favourite repo name too, and Sigma will create it if it does not exist. (However, Sigma has a known glitch when an "empty" repo (i.e. one that does not have a master branch) is encountered, so if you have a brand new repo, make sure that you have at least one commit on the master branch; the easiest way is to create a Readme, which can be easily done with one click at repo creation.)

commit dialog

Once saving is complete, Sigma will automatically build your project, and open up a deployment summary pop-up showing everything that it would deploy to your AWS account with regard to your brand new S3 thumbnail generator. Some of the names will look gibberish, but they will generally reflect the type and name of the deployed resource (e.g. s3MyAwesomeBucket may represent a new S3 bucket named my-awesome-bucket).

build progress in status bar

deployment changes summary

Review the list (if you dare) and click Deploy. The deployment mechanism will kick in, displaying a live progress bar (and a log view showing the changes taking place in the underlying CloudFormation stack of your project).

deployment in progress

Once the deployment is complete, your long-awaited thumbnail generator lambda is ready for testing! Just upload a JPG or PNG file to the source bucket you chose (via the S3 console, or via an aws s3 cp if you are more like me), and marvel at the thumbnail that would pop up in your destination bucket within a matter of seconds!

If you don't see anything interesting in the destination bucket (after a small wait), you would be able to check what went wrong, by checking the lambda's execution logs just like in the case of any other lambda; we know it's painful to go back to the AWS consoles to do this, and we hope to find a cooler alternative to that as well, pretty soon.

If you want to make the generated thumbnail public (as I said in my previous article, what good is a private thumbnail?), you don't have to run around reading IAM docs, updating IAM roles and pulling your hair off; simply click the S3 operation edit icon against the s3.putObject call, select the "ACL to apply to the object" parameter as public-read from the drop-down, and click "Deploy" to go through another save-build-deploy cycle. (We are already working on speeding up these "small change" deployments, so bear with us for now :) ) Once the new deployment is complete, in order to view any newly generated thumbnails, you can simply enter the URL http://<bucketname>.s3.amazonaws.com/resized-<original image name> into your favourite web browser and press Enter!

making thumbnails public: S3 pop-up

Oh, and if you run into anything unusual—a commit/build/deployment failure, an unusual error or a bug with Sigma itself— don't forget to ping us via Slack - or post an issue on our public issue tracker; you can do it right within the IDE, using the "Help" → "Report an Issue" menu item. Same goes for any improvements or cool features that you would like to see in Sigma in the future: faster builds and deployments, ability to download the build/deployment artifacts, a shiny new set of themes, whatever. Just let us know, and we'll add it to our backlog and give it a try in the not-too-distant future!

Okay folks, time to go back and start playing with Sigma, while I write my next blog post! Stay tuned for more from SLAppForge!