Look, look! EI on LXC!

Containers have landed. And they are conquering the DevOps space. Fast.

Enterprise Integration has been leading and driving the business success of organizations for ages.

Now, thanks to AdroitLogic IPS, you can harness the best of both worlds - all the goodies of enterprise integration (EI), powered by the flexibility of Linux containers (LXC) - for your rapidly scaling, EI-hungry business and enterprise!

In case you had already looked at IPS, you might have noticed that it only offers a VirtualBox-based, single-instance evaluation distribution - not the most realistic way to try out a supposedly highly-available, highly-scalable integration platform.

But this time we have something "real" - something you can try out on your own K8s (or K8s-compatible; say OpenShift) cluster. It offers much better scalability - you can deploy our high-performance UltraESB-X instances across up to 20 physical nodes, in unlimited numbers (just ping us if you need more!) - and flexibility - the underlying K8s infra is totally under your control; for upgrades, patches, HA, DR and whatnot.

But the best part is that your UltraESB-X enterprise integrator instances would be running right inside your K8s network, meaning that you can easily and seamlessly integrate them with all your existing stuff: services (or microservices), queues, CI/CD pipelines, messaging systems, management and monitoring mechanisms, you name it.

You can run the new IPS installer from any machine that has SSH access to the cluster - even from within the cluster itself. The process is fairly simple: just

• download and extract the installer (~ 18 KB),
• copy your client key into the node directory (you can either grab one via our evaluation form or use any other one that you have obtained previously from AdroitLogic; maybe one for UltraESB-X, UltraStudio or IPS (Mac/Linux)),
• edit config.sh with the necessary configurations (generally just the user-host combinations (user@host) of the master and worker nodes would suffice), and
• run deploy.sh!

If all goes well, the mission will go like this:

               ***************************************
                      Welcome to IPS Installer!
               ***************************************

Loading configurations...

++ MASTER=ubuntu@ip-1-2-3-4
++ NODES=(ubuntu@ip-5-6-7-8 ubuntu@ip-9-0-1-2)
++ SSH_ARGS='-i /path/to/aws/key.pem'
++ DB_IN_CLUSTER=true
++ DB_URL='jdbc:mysql://mysql.ips-system.svc.cluster.local:3306/ips?useSSL=false'
++ DB_USER=ipsuser
++ DB_PASS='7h1Zl$4v3RyI95e~CUr#*@m0R+'
++ DB_NODE=
++ ES_ENABLED=false
++ ES_IN_CLUSTER=true
++ ES_HOST=elasticsearch.ips-system.svc.cluster.local
++ ES_PORT=9300
++ ES_NODE=
++ DOCKER_REPO=adroitlogic
++ DOCKER_TAG=17.07.2-SNAPSHOT
++ set +x

Checking configurations...

NOTE: DB_NODE was not specified, defaulting to ip-5-6-7-8.
NOTE: ES_NODE was not specified, defaulting to ip-9-0-1-2.
Configurations checked. Looks good.

IPS will download required Docker images into your cluster
(~550 MB, or ~400 MB if you have disabled statistics).

Agree? yes


Starting IPS installation...

IPS needs to download the MySQL Java client library (mysql-connector-java-5.1.38-bin.jar)
in order to proceed with the installation.
Please type 'y' or 'yes' and press Enter if you agree.
If you are curious to know why we do it this way, check out
https://www.mysql.com/about/legal/licensing/oem/#3.

Agree? 

At this point you can either accept the proposal (obvious choice) or deny it (in which case the installer would fail).

Should you choose to accept it:

Agree? yes


Starting IPS installation...

Preparing ubuntu@ip-5-6-7-8...
Connection to ip-5-6-7-8 closed.
client.key.properties                                                                                                                                                            100%   53     0.1KB/s   00:00
license.conf.properties                                                                                                                                                          100%   78     0.1KB/s   00:00
license.key.properties                                                                                                                                                           100%   53     0.1KB/s   00:00
mysql-connector-java-5.1.38-bin.jar                                                                                                                                              100%  961KB 960.9KB/s   00:00
Successfully prepared ubuntu@ip-5-6-7-8

Preparing ubuntu@ip-9-0-1-2...
Connection to ip-9-0-1-2 closed.
client.key.properties                                                                                                                                                            100%   53     0.1KB/s   00:00
license.conf.properties                                                                                                                                                          100%   78     0.1KB/s   00:00
license.key.properties                                                                                                                                                           100%   53     0.1KB/s   00:00
mysql-connector-java-5.1.38-bin.jar                                                                                                                                              100%  961KB 960.9KB/s   00:00
Successfully prepared ubuntu@ip-9-0-1-2

configserver-rc.yaml                                                                                                                                                             100% 1960     1.9KB/s   00:00
configserver-svc.yaml                                                                                                                                                            100%  415     0.4KB/s   00:00
elasticsearch-rc.yaml                                                                                                                                                            100% 1729     1.7KB/s   00:00
elasticsearch-svc.yaml                                                                                                                                                           100%  418     0.4KB/s   00:00
ips-admin.yaml                                                                                                                                                                   100% 1093     1.1KB/s   00:00
ips-stats.yaml                                                                                                                                                                   100%  684     0.7KB/s   00:00
ipsweb-rc.yaml                                                                                                                                                                   100% 5023     4.9KB/s   00:00
ipsweb-svc.yaml                                                                                                                                                                  100%  399     0.4KB/s   00:00
mysql-rc.yaml                                                                                                                                                                    100% 1481     1.5KB/s   00:00
mysql-svc.yaml                                                                                                                                                                   100%  360     0.4KB/s   00:00
namespace "ips-system" created
namespace "ips" created
clusterrole "ips-node-stats" created
clusterrolebinding "ips-node-stats" created
clusterrole "ips-stats" created
clusterrolebinding "ips-stats" created
clusterrole "ips-admin" created
clusterrolebinding "ips-admin" created
replicationcontroller "mysql" created
service "mysql" created
replicationcontroller "configserver" created
service "configserver" created
replicationcontroller "ipsweb" created
service "ipsweb" created

IPS installation completed!
The IPS dashboard will be available at https://ip-5-6-7-8:30080 shortly.

You can always reach us at
    info@adroitlogic.com
or
    https://www.adroitlogic.com/contact/

Enjoy! :)

That's it! Time to fire up the dashboard and get on with it!

A few things to note, before you rush:

• the hostnames/IP addresses used in config.sh should be the same as those being used as node names on the K8s side. Otherwise the IPS components may fail to recognize each other and the master. For now, an easy trick is to directly use the K8s node names for MASTER and NODES parameters (oh, and don't forget DB_NODE and ES_NODE!), and add host entries (maybe /etc/hosts on the installer machine) pointing those names to the visible IP addresses of the actual host machines; until we make things more flexible in the not-too-distant future.
• Docker images for IPS management components will start getting downloaded on demand, as and when they are defined on the K8s side. Hence it may take some time for the system to stabilize (that is, before you can access the dashboard).
• Similarly, the UltraESB-X Docker image will be downloaded on a worker node only when the first ESB instance gets scheduled on that node, meaning that you might observe slight delays during the first few ESB cluster deployments. If necessary, you can avoid this by manually running docker pull adroitlogic/ips-worker:17.07.2-SNAPSHOT on each worker node.

With the new distribution, we have also allowed you to customize the place where you store your IPS configurations (MySQL) and statistics (Elasticsearch): you can either set DB_IN_CLUSTER (or ES_IN_CLUSTER) to false and specify an external MySQL DB (or ES server) using DB_HOST, DB_PORT, DB_USER and DB_PASS (or ES_HOST and ES_PORT), or set it to true and specify a node name where MySQL (ES) should be deployed as an in-cluster pod, using DB_NODE (ES_NODE). Using an external MySQL or ES instance may be handy for cases where your cluster has limited resources (especially memory) and you want to maximally utilize them for running ESB instances rather than allocating them for infrastructure components.

Additionally, now you can also disable some non-essential features of IPS, such as statistics, at installation itself; just set ES_ENABLED to false, and IPS will skip the installation of an ES container and also stop the collection of ESB statistics at runtime. This can be really handy if you are running ESBs in a resource-constrained environment - disabling ES can bring down the per-ESB startup memory footprint from 700 MB right down to 250 MB! (We are already working on a leaner statistics collector based on the lean and sexy ES REST client - along with some other cool improvements - and once it is out, you will be able to run stats-enabled ESBs at under 150 MB memory.)

The new release is so new that we barely had the time to write the official docs for it - but all the existing docs, including the user guide and samples are all applicable to it, with some subtle differences:

• The ESB Docker image (ips-worker) uses a different tag - a steaming hot 17.07.2-SNAPSHOT, instead of the default 17.07.
• In samples, while you previously had to use the host-only address of the VM-based IPS node for accessing deployed services, now you can do it in a more "natural" way - by using the hostname of any node in the K8s cluster, just as you would do with any other K8s-based deployment.

For those of you who are starting from scratch, we have included a tiny guide to get you started with kubeadm - derived from the official K8s docs - that would kick-start you with a fully-managed K8s cluster within minutes, on your favourite environment (bare-metal or cloud). We also ship a TXT version inside the installer archive, in case you want to read it offline.

And last but not the least, if you don't like what you see (although we're pretty sure that you will!), you can purge all IPS-related things from your cluster (:sad_face:) with another single command, teardown.sh:

++ MASTER=ubuntu@ip-1-2-3-4
++ NODES=(ubuntu@ip-5-6-7-8 ubuntu@ip-9-0-1-2)
++ SSH_ARGS='-i /path/to/aws/key.pem'
++ DB_IN_CLUSTER=true
++ DB_URL='jdbc:mysql://mysql.ips-system.svc.cluster.local:3306/ips?useSSL=false'
++ DB_USER=ipsuser
++ DB_PASS='7h1Zl$4v3RyI95e~CUr#*@m0R+'
++ DB_NODE=
++ ES_ENABLED=false
++ ES_IN_CLUSTER=true
++ ES_HOST=elasticsearch.ips-system.svc.cluster.local
++ ES_PORT=9300
++ ES_NODE=
++ DOCKER_REPO=adroitlogic
++ DOCKER_TAG=17.07.2-SNAPSHOT
++ set +x
Starting IPS tear-down...

Tearing down master...
namespace "ips-system" deleted
namespace "ips" deleted
clusterrole "ips-node-stats" deleted
clusterrole "ips-admin" deleted
clusterrole "ips-stats" deleted
clusterrolebinding "ips-node-stats" deleted
clusterrolebinding "ips-admin" deleted
clusterrolebinding "ips-stats" deleted
Successfully tore down master

Tearing down ubuntu@ip-5-6-7-8...
Connection to ip-5-6-7-8 closed.
Successfully tore down ubuntu@ip-5-6-7-8

Tearing down ubuntu@ip-9-0-1-2...
Connection to ip-9-0-1-2 closed.
Successfully tore down ubuntu@ip-9-0-1-2

IPS tear-down completed!

Enough talking, time to jump-start your integration - this time, on containers!

Gracefully Shutting Down Java in Containers: Why You Should Double-Check!

Gracefulness is not only an admirable human quality: it is also a must-have for any application program, especially when it is heaving the burden of mission-critical domains.

UltraESB has had a good history of maintaining gracefulness throughout its runtime, including shutdown. The new UltraESB-X honoured the tradition and implemented graceful shutdown in its 17.07 release.

When we composed the ips-worker Docker image for our Integration Platform (IPS) as a tailored version of UltraESB-X, we could guarantee that ESBs running in the platform would shut down gracefully--or so we thought.

Unfortunately not.

As soon as we redeploy or change the replication count of a cluster, all ESB instances running under the cluster would terminate (and new instances get spawned to take their place). The termination is supposed to be graceful; the ESBs would first stop accepting any new incoming messages, and hold off the internal shutdown sequence for a few seconds until processing of in-flight messages gets completed, or a timeout ends the holdoff.

On our Kubernetes-based mainstream IPS release, we retrieve logs of ESB instances (pods) via the K8s API as well as via a database appender so that we can analyze them later. In analyzing the logs, we noticed that we were never seeing any ESB shutdown logs, no matter how big the log store has grown. It was as if the ESBs were getting brutally killed as soon as the termination signal was received.

To investigate the issue, I started off with a simplified Java program: one that registers a shutdown hook--the world-famous way of implementing graceful shutdown in Java, which we had utilized in both our ESBs--and keeps running forever, printing some text periodically (to indicate the main thread is active). As soon as the shutdown hook is triggered, I interrupted the main thread, changed the output to indicate that we are shutting down, and let the handler finish after a few seconds (consistent with a "mock" graceful shutdown).

class Kill {

	private static Thread main;

	public static void main(String[] a) throws Exception {

		Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
			public void run() {
				System.out.println("TERM");
				main.interrupt();
				for (int i = 0; i < 4; i++) {
					System.out.println("busy");
					try {
						Thread.sleep(1000);
					} catch (Exception e) {}
				}
				System.out.println("exit");
			}
		}));

		main = Thread.currentThread();
		while (true) {
			Thread.sleep(1000);
			System.out.println("run");
		}
	}
}

Testing it is pretty easy:

javac Kill.java
java Kill

While the program keeps on printing:

run
run
run
...

press Ctrl+C to see what happens:

...
run
run
^CTERM
busy
Exception in thread "main" java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at Kill.main(Kill.java:22)
busy
busy
busy
exit

Looks good.

That done, converting this into a fully-fledged Docker container took only a few minutes, and the following Dockerfile:

FROM openjdk:8-jre-alpine
ADD Kill*.class /
ENTRYPOINT ["java", "Kill"]

docker build -t kill:v1 .

Next I ran a container with the new image:

docker run -it --rm kill:v1

which gave the expected output:

run
run
run
...

Then I sent a TERM signal (which maps to Ctrl+C in normal jargon, and is the default trigger for Java's shutdown hook) to the process, using the kill command:

# pardon the fancy functions;
# they are quite useful for me when dealing with processes

function pid() {
    ps -ef | grep $1 | grep -v grep | awk '{print $2}'
}

function killsig() {
    for i in pid $2; do
        sudo kill $1 $i
    done
}

alias termit='killsig -15'

# with all the above in place, I just have to run:
termit Kill

As expected, the shutdown hook got invoked and executed smoothly.

Going a step further, I made the whole thing into a standalone K8s pod (backed by a single-replica Deployment):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kill
spec:
  selector:
    matchLabels:
      k8s-app: kill
  template:
    metadata:
      labels:
        k8s-app: kill
    spec:
      containers:
      - name: kill
        image: kill:v1

and tried out the same thing, this time by zeroing-out spec.replicas (same as we do it in IPS) via the kubectl edit deployment command, instead of a manual kill -TERM:

kubectl edit deployment kill

# vi is my default editor
# set "replicas" to 0 (line 20 in my case)
# <ESC>:wq<ENTER>

while having a console tail of the pod in a separate window:

# fancy stuff again

function findapp() {
    kubectl get pod -l k8s-app=$1 -oname | cut -b 6-;
}

function klog() {
    kubectl logs -f findapp $1;
}

# the final command
klog kill

showing the output:

run
run
...
run
TERM
busy
Exception in thread "main" java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at Kill.main(Kill.java:22)
busy
busy
busy
exit

Damn, it still shuts down gracefully!

So what's wrong with my ips-worker?

Just to verify, I got a single-replica cluster running on IPS, manually changed the image (spec.template.spec.containers[0].image) and startup command (spec.template.spec.containers[0].command) of the K8s deployment via kubectl edit (keeping all other factors--such as environmental variables and volume mounts--unchanged), and tried out the same zero-out sequence;

Same result! Graceful shutdown!

Then it occurred to me that, while my kill container simply uses a java Kill command, ips-worker uses a bit more complicated command:

/bin/sh -c <copy some files> && <run some custom command> && <run ultraesb-x.sh>

where, in the last part, we construct (with a specially fabricated classpath, and some JVM parameters) and execute a pretty long java command that starts up the UltraESB-X beast.

So ultimately, the final live command in the container boils down to:

/bin/sh -c <basepath>/ultraesb-x.sh

Hence I tried a shell command on my kill container, by slightly changing the Dockerfile:

by slightly changing the Dockerfile:

FROM openjdk:8-jre-alpine
ADD Kill*.class /
# note the missing brackets and quotes, so that the command gets the default /bin/sh -c prefix
ENTRYPOINT java Kill

and yay! Graceful shutdown was no more. The Java process got killed brutally, on Docker (docker stop) as well as in K8s (replica zero-out).

Investigating further, I was guided by Google to this popular SE post which basically said that the shell (sh) does not pass received signals to its child processes by default. The suggested alternative was to run the internal command as an exec which would basically replace the parent process (sh) with the child (java, in case of kill):

FROM openjdk:8-jre-alpine
ADD Kill*.class /
ENTRYPOINT exec java Kill

For kill, that did the trick right away.

For ips-worker things were a bit different, as there were two levels of invocation: the container's command invoking a chain of commands via /bin/sh -c, and the built-in ultraesb-x.sh invoking the ultimate java command. Hence I had to include exec at two places:

Once at the end of the command chain:

/bin/sh -c \
<copy some files> && \
<run some custom command> && \
exec <basepath>/ultraesb-x.sh

And again at the end of ultraesb-x.sh:

# do some magic to compose the classpath and other info for ESB startup

exec $JAVA_HOME/bin/java <classpath and other params>

Simple as it may seem, those two execs were enough to bring back graceful shutdown to ips-worker, and hence to our Integration Platform.