File Transfer, MFT and Beyond: The Story of AS2

Originally written for The AS2 Gateway Blog; Jun 17, 2020

This is a work of fiction. The incidents portrayed herein may have taken place under different settings, in a different order, or even not taken place at all; however, it will not refute the origin and evolution of Applicability Statement 2, more commonly known as AS2, the most popular secure B2B document/EDI exchange protocol in the known universe.

The Beginning: File Transfer, the Primordial Soup

Earliest files were paperbacked, and dispatched around via postal and courier services. They took days - or hours at best - to reach their destinations. Needless to say, a major bottleneck in rapidly growing B2B trade ecosystems.

Digital File Transfer

Then came computers and the internet, and things improved dramatically. (There were probably many more intermediates; fax, email, etc.; but we'll skip them for brevity.)

Now the transfers took just minutes - sometimes even seconds - instead of hours or days. Accuracy also improved greatly - no more deliveries to the wrong doorstep. Protocols like FTP and email soon became the de-factos for digital data exchange and file transfer.

But still, not everything was solid and foolproof:

Acknowledgement and Integrity

There was no receipt or acknowledgement/confirmation; you could send a PO, wait several days for the invoice, and finally phone them to find out they have somehow missed the file - still lying in a corner on their FTP server.

Also, the network did not guarantee integrity (delivery "as a whole"); you could have sent a 1000-row file, and the last 10 - or even 10 randoms from all over the place - could have gotten "lost". Even worse, a random flip of a few bytes could convert some five-digit price tag into a nine-digit one - I won't even start on the consequences.

Receipts and Hashes

So people and systems started to acknowledge these transfers with digital receipts; if you sent a file and received an acknowledgement (possibly another file) in return, you had a guarantee (non-repudiation) that the trading partner (or his system) has seen the file and taken it up for processing.

Now that there was two-way communication, there was also a way to verify that the receiver got exactly what the sender sent out (integrity); the receiver would calculate a hash (a finite-length summary; think of finding the "lucky number" from your birthday) of the received file, and send it back in the receipt. The sender could then do the same on the original file she sent, and verify that the two match - which would mean the content on both sides is practically identical (down to just a subatomic mergin of error).

Two problems solved; but still, there were more serious ones lurking around.

Eavesdropping, Tampering and Spoofing

File content was flowing through the (inherently unsecure) internet, in cleartext. This meant, any of your adversaries could eavesdrop on the files you are sending - and worse, even modify (tamper) the data being transmitted, to sabotage or manipulate your transactions.

True, you are already exchanging and validating the hash for the sent/received content; but a decent man-in-the-middle could easily pause the original data flow, do his modification and generate a new hash that would match the modified data.

(Or, if he had complete control over the data flow, he could simply intercept the partner's receipt as well, replace the hash with the old one, and relay it back to you, the unsuspecting victim - who would naturally assume everything went along just fine.)

On the other hand, when it comes to receiving files or receipts back from your partner, you had to somehow make sure that it was actually your partner sending those data in the first place - not something spoofed by a malicious third party.

Digital Signature: for Tamper-proof Integrity

What we needed here was simple: a way for you (sender) to place a "seal" on the file (very much like the wax seals from the old snail-mail days) and for your partner (recipient) to verify that the seal was not broken while things were in transit.

So the digital signature was born. Basically you take the hash of the file (as before), encrypt it with your private key, and send it alongside; and publish the corresponding public key (certificate) so that other people - usually your partners - can decrypt (verify) that blurb of data upon receipt.

This was a rather smart move! It achieved many goals:

Since only you had the private key, the partner now had a strong guarantee that the file came from you - and you couldn't refute that either, even if you wanted to; authentication and non-repudiation.
The "safely transmitted" hash guaranteed the file's intactness; integrity.
Even if there was no anti-tamper encryption (next section) in place, if anyone managed to modify the file and generate a new matching hash, they would not be able to generate a verifiable signature out of it - without breaking into your private space and stealing your very own private key.

Good; now if we could just make the file unreadable to prying eyes, in transit...

Encryption: for Invisibility (Confidentiality)

As the title suggests, encrypting the file content makes it invisible (more correctly, incomprehensible) while it flows through the internet. Which means there is no room for the adversary to read or modify the file content.

Of course there had already been 'secure' protocols like SCP, SFTP and FTP/S that would transfer encrypted content (ciphertext) instead of cleartext; however, you still had to make sure that you were connecting and sending to the correct remote partner/system; jumping over traps like DNS spoofing.

But with AS2 encryption, even if you connected to a wrong server, your content was encrypted using a certificate (public key) that was pre-shared with you by the real partner - and not the certificate of the currently connected server. So, even if you did accidentally send the file to a hacker, they could not possibly decrypt it without access to the partner's own secret private key - which never comes out in the open, throughout the whole process.

AS2: there you are!

What we (or rather, many ingenious people over several decades) invented so far - adds up to the AS2 protocol - which happens to be the most popular secure file transfer protocol today. Especially after Walmart the Giant mandated it for all its trading partners since 2002.

You can learn more about the protocol from its official RFC 4130 document, which is very informative and fairly simple to understand.

Having said that, where are we now?

All set; file transfer is secure! But...

If you scroll up towards the "Digital file transfer" section you would realize that, while we have achieved a remarkable level of security, AS2 has also added a lot of overhead and complexity to our business flow:

AS2: secure, but complex!

Let's see what it takes:

To send one batch of files

share the connectivity data and encryption (and signature) certificates with your trading partner (thankfully, this is a one-time task - for each trading partner, that is)
combine the file(s) you wish to transfer, into one big MIME message
compress the MIME (to reduce its size and save transfer time/bandwidth; could be significant depending on the file data)
compute the hash of the data blurb at hand
sign the blurb with your private key (and keep it secure)
encrypt the whole thing with your partner's public key
transfer the encrypted data over to your partner's system
wait for the receipt (MDN) from the partner
check the signature of the MDN against the partner's public key (to make sure it wasn't forged by someone else - and hasn't been tampered with)
process the MDN to see if everything went fine - or whether there were any errors on the partner's end (opening up a whole new pathway for troubleshooting)
check the hash (message integrity code, MIC) sent in the MDN against what you calculated in the beginning, to make sure that the partner received the same data that you sent out - untampered
if the MDN was asynchronous, acknowledge it properly to close out the connection (and with it, the transaction)

And when receiving a batch of files:

The situation is as much complicated:

share the connectivity data and certificates (one-time per partner)
keep on listening or checking for incoming messages from your partner
when one is received, accept and load the incoming blurb of data
decrypt the data using your own private key
extract out and verify the signature part using your partner's public key - to ensure integrity
decompress the data part to get to the MIME of files
decompose the MIME into original files sent by your partner
compute the MIC over the actual data
compose an MDN with the MIC, and any errors you encountered during the process so far
sign the MDN with your private key - for authenticity
send the MDN back to your partner, and ensure they received it

Obviously, too much of a technical burden for an already-complicated business flow, and a team that is already overwhelmed with a thousand and one other business-critical tasks and needs.

There are tools like the openssl suite that can take good care of individual steps; but chaining them up to fulfill the overall flow is not very trivial or feasible, even for a tech-savvy team; and most business teams cannot afford such a subdivision of techies in the first place.

Managing the transfers: more overhead

While it solves the security aspect and provides an end-to-end guarantee of the file transfer, AS2 by itself does not fully solve the caveats of file transfer, especially being over the unreliable internet:

If the transfer fails mid-way - or the partner system reports an error, confirming that it couldn't get hold of the files - you need to replay the transfer.
If the transfer itself is successful but you don't receive a receipt (MDN) after a considerable wait, a replay may be required as well.

Furthermore:

As your business grows, it should be able to automate the AS2 transfer aspect as well - rather than having to manually submit each file or manually inspect every MDN receipt.
Going with the ever-growing trend of digital transformation, file transfer also needs to become a well-streamlined piece of the overall business data flow in your organization.
Setting up and juggling with multiple trading partners, with dozens of certificates, endpoints and pieces of configuration, can soon become daunting and error-prone.
Auditing and keeping track of transfer history and failures could be important for QoS/reliability assessments as well as business-level audits.

And that is why many new as well as well-established businesses are turning towards:

Managed AS2 and MFT Solutions

Managed File Transfer collectively refers to addressing the pain points we discussed so far:

All the security and end-to-end delivery guarantees of AS2 (or another MFT protocol)
Easy and simplified configuration management for multiple trading partners and local identities (stations)
Enhanced delivery and visibility, such as automatic retries and notifications in case of permanent failures
Enterprise-ready: ability to easily integrate with your existing business systems via popular mechanisms like APIs, queues, webhooks, secondary file transfer protocols, cloud services etc.

Such MFT systems can be:

on-premise (deployed on your own local data center) or cloud hosted (on a public cloud platform like AWS EC2)
dedicated or shared (multi-tenant)

The choice depends mainly on:

your expected file exchange load a.k.a. volume (distribution and numbers per minute/hour/day, average size of one file, etc.)
the quality of service (QoS) that you expect (for example, will a couple minutes' downtime - fairly common in shared, multi-tenant SaaS platforms - critically affect your business? If so, a dedicated deployment with a fail-over/high-availability (HA) server is the way to go.)
how deeply you wish to integrate with or customize the platform; a regular SaaS would no longer make sense beyond a certain point - in contrast, a dedicated deployment would always offer the highest and fastest level of adaptability
your organization policies, and security/compliance requirements (e.g. PCI, SOC2, GDPR)
and, of course, your budget

Where to Go from Here

AS2 Gateway is a simple yet mature AS2-based MFT platform, nurtuted and trusted over a decade of production use by customers all over the globe.

In addition to an intuitive, email-like interface for exchanging files with your trading partners, it offers automation mechanisms like SFTP and a REST API that will closely integrate with your own order processing system and business workflow. More integration methods like webhooks and cloud storage like AWS S3 are already on the way.

Getting started with AS2 Gateway SaaS cloud hosted platform is pretty easy; there is comprehensive documentation, demo videos and troubleshooting references to guide you every step of the way. Besides, the friendly AS2 Gateway team is ready to help you at any hour of the day.

If your use case calls for a dedicated installation, with higher messaging capacity, cloud or on-premise hardware, and custom integrations with your own systems, it is just a quick call or form-fill away.

Wednesday, June 24, 2020