Project-X S01E01: Pilot

P.S.: Okay, while the Pilot is supposed to arouse curiosity and keep you on the edge of the seat for S01E02, I'm not sure how well I've done that—rereading what I just completed writing. Anyway, see for yourself!

In February 2017, something happened. Something that has never been seen, and rarely been heard, ever before.

Project-X.

No, not the movie! That was in 2012!

Engine. Messaging Engine.

A lean (~9 MB), fast (yup, benchmarked) and reliable messaging engine.

But not just a messaging engine.

The original authors supposed it to be an Enterprise Service Bus (ESB). But, over time, it evolved.

Into a simple yet formidable integration middleware product.

Capable of connecting the dots: you, your customers, your partners, and the wide, wide world (WWW).

Proven by powering the cores of many other integration solutions, including the famous B2B AS2 trading platform.

And, most importantly, something that can help you—and your company—tackle its integration challenges at a fraction of the time, effort and cost. Something that would allow you to draw your solution rather than coding or configuring it. Something that would make your PoC as good as your final draft, as they would essentially be the same.

Something that would make you enjoy integration, rather than hating it.

While you can always explore our official documentation to unravel how all this works (yup, I heard that yawn :)) the following couple of facts is all you need to know in order to grab Project-X by its horns:

Project-X deals with messages, or distinct and quantifiable bits of information (a clock tick, a HTTP request, an event/message over JMS/Kafka, a file dropped into your SFTP folder, a neutron emitted from a U-235, an interplanetary collision,... you name it).

The beauty is the fact that almost every bit of interaction in an integration (connecting-the-dots) scenario can be mapped into a message. Care to see some examples?

• "A consumer is calling your API" maps into "a HTTP request"
• "Your customer just bought 3 techno-thrillers from your ebook store" could map to "an event over Kafka"
• "Your partner just sent you an invoice" could map into "a file dropped into your SFTP folder"

Being the expert on messages, Project-X can take up the rest.

Project-X can consume messages from an insanely wide variety of sources. Ingress connectors bring in these messages into the Project-X engine. We already have connectors for all of the above examples, except maybe for the last (left as an exercise for the reader).

Project-X can emit messages into an equally wide array of destinations over different media. Egress connectors are the ones that do this.

In between consumption and emission, a message can go through all sorts of weird experiences, which could consist of transormation, extraction, enrichment, conditional branching, looping, split-aggregation, throttling, exceptions,... and a lot more of other unimaginable stuff. Processing elements perform all this magic.

(By the way, This document deals with all the gruesome details, in case you are interested.)

An ingress connector, a chain of processing elements and an egress connector, together make up an integration flow that represents everything that a message is destined to go through, given that it is sucked in by the ingress connector. A simple analogy is a conveyor belt on a production line, where something comes in and something (similar or different) comes out. Forget not, however, that depending on your requirement, an integration flow can be made as complex, rich and flexible as you like; your imagination being the only limit.

Related integration flows are often bundled into a single integration project (yup, we love that word "integration"), which can be deployed as a single unit of functionality in a Project-X engine such as UltraESB-X.

Project-X keeps track of each and every message flying through the integration flows in every single project that is deployed in its runtime, and fully manages the lifecycle of each message (including allocating its required resources, routing it to the correct connector, pumping it through the flow, handling failures, gathering metrics, and cleaning up after its death/completion).

On top of all this, Project-X has its own bag of tricks, goodies and other cool stuff:

For you, the tech-savvy:

• Configurable transactionality (end-to-end or subsectional, as required)
• Project-level dependency management and class spaces (stuff in one project can never mess with stuff in another project); via the famous hot-swap classloader
• Graceful engine lifecycle (no message or event is left dead in its tracks, may it be startup or shutdown)
• Efficient queue management via scale-first execution
• Many goodies from the legendary UltraESB, including zero-copy proxying, garbage-free payload handling, thread-local transactions and more, rolled into one

For you, developers and architects:

• A five-minute prototyping platform for your next integration project, whose thin-slice PoC outcome would eventually evolve into a fully-blown production deployment
• An intuitive, DIY integration surface with the familiar notions of messages, events, connectors and flows, instead of painful XML configurations
• A warehouse of ready-made connectors and processing elements to choose from
• A simple and flexible way to create your own connectors and processors, and share/reuse them across projects (so you never have to reinvent the wheel across your colleagues, teams or departments... or even companies)
• A super-cool IDE where you can graphically compose your integration solution (drag-and-drop, set-up and wire-together) and test, trace and debug it right away

For you, deployers and sysadmins:

• A pure Java-based runtime (no native libs, OSGi, voodoo or animal sacrifices)
• Pluggable, REST-based, secure management API for remote administration
• Pluggable analytics via Elasticsearch metrics collectors
• Ready-made runtime bundled with the IDE for seamless dev-test cycles
• Quick-and-dirty testing with ready-made Docker images: choose between slim and rich
• A production bundle complete with a daemon (service-compatible) deployment, and in-built statistics and management servers
• A tooling distribution for management operations: a CLI for engine governance, and ZooKeeper client/server bundles for clustering
• A fully-fledged management console for tracking, monitoring and managing your deployment, complete with statistics, alerting and role-based access control
• A fully-fledged containerized deployment platform, that can run in-cloud or on-premise, for deploying and managing Project-X clusters with just a few mouse clicks (and keystrokes)

To be continued...

(Keep in touch for S01E02!)