Wednesday, May 4, 2016

Harmonizing UltraESB and FTP/S: Troubleshooting Connectivity

The UltraESB has built-in capability of communicating with FTP, FTP/S, SFTP and local filesystem (file://) endpoints. While setting up the other file transports might be easy, FTP/S can be problematic, especially if you start from a fresh FTP/S installation.

Assuming a local FTP/S server setup (e.g. vsftpd) with a user ftp having password ftp and home directory /srv/ftp (quoting vsftpd default settings), you might encounter a few problems when trying to submit files to an endpoint URL such as ftps://ftp:ftp@localhost/srv/ftp/, on the UESB side.

These usually occur at FTP level, so in order to find the actual causes you will have to enable TRACE logging on the org.adroitlogic.ultraesb.transport.file category in $ULTRA_HOME/conf/log4j.properties:

log4j.category.org.adroitlogic.ultraesb.transport.file=TRACE

With TRACE enabled as above, the UESB log will contain a trace of all FTP commands being executed during the transfer attempt, as:

2016-05-04 22:57:55,043 [-] [primary-1] [system] [000000I] DEBUG FTPSConnector Connecting over FTPS to URL : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no

220 (vsFTPd 3.0.3)
AUTH TLS
234 Proceed with negotiation.
PROT P
200 PROT now Private.
USER ftp
...

If you see 530 Anonymous sessions may not use encryption. (usually followed by a stacktrace on the UESB):

...
234 Proceed with negotiation.
PROT P
200 PROT now Private.
USER ftp
530 Anonymous sessions may not use encryption.
QUIT
221 Goodbye.
2016-05-04 22:57:55,381 [-] [primary-1] [system] [000000W]  WARN FTPSConnector Login failed for FTPS server : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
PORT 127,0,0,1,179,254
2016-05-04 22:57:55,385 [-] [primary-1] [system] [000000W]  WARN FTPConnector Error copying file to URL : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
org.apache.commons.net.ftp.FTPConnectionClosedException: Connection closed without indication.
	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:290)
	at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:479)
	at org.apache.commons.net.ftp.FTPSClient.sendCommand(FTPSClient.java:535)
...

you would want to allow anonymous SSL sessions on your vsftpd.conf:

allow_anon_ssl=YES

(Note: Make sure you restart vsftpd after every such change!
service vsftpd restart # as root)

If the issue is 522 SSL connection failed: session reuse required:

...
234 Proceed with negotiation.
PROT P
200 PROT now Private.
USER ftp
331 Please specify the password.
PASS ftp
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (127,0,0,1,161,222).
STOR /srv/ftp/ftps/send.msg
150 Ok to send data.
522 SSL connection failed: session reuse required
2016-05-04 22:59:07,418 [-] [primary-1] [system] [000000I] DEBUG FTPConnector FTP copy failed to : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
2016-05-04 22:59:07,429 [-] [primary-1] [system] [000000W]  WARN FileTransportSender Error sending message via file transport sender to : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no

adding/updating require_ssl_reuse=NO in vsftpd.conf would resolve it.

On the other hand, if authentication fails mid-way in the SSL handshake process:

220 (vsFTPd 3.0.3)
AUTH TLS
234 Proceed with negotiation.
2016-05-04 23:05:26,907 [-] [primary-2] [system] [000000W]  WARN FTPSConnector Error connecting over FTPS to URL : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
...
	at org.adroitlogic.ultraesb.transport.file.FTPSConnector.(FTPSConnector.java:62)
	at org.adroitlogic.ultraesb.transport.file.FileSystemConnectorFactory.createConnector(FileSystemConnectorFactory.java:54)
...
	at org.adroitlogic.ultraesb.core.work.SimpleQueueWorkManager$1.run(SimpleQueueWorkManager.java:306)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2016-05-04 23:05:26,909 [-] [primary-2] [system] [000000W]  WARN FTPConnector Error copying file to URL : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
java.net.SocketException: Socket closed
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
...
	at org.apache.commons.net.ftp.FTP.__send(FTP.java:501)
	at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:475)
...
	at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:1795)
	at org.adroitlogic.ultraesb.transport.file.FTPConnector.put(FTPConnector.java:169)
	at org.adroitlogic.ultraesb.transport.file.FileTransportSender.internalSend(FileTransportSender.java:80)
...
	at org.adroitlogic.ultraesb.core.work.SimpleQueueWorkManager$1.run(SimpleQueueWorkManager.java:306)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2016-05-04 23:05:26,919 [-] [primary-2] [system] [000000W]  WARN FileTransportSender Error sending message via file transport sender to : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no

you might require a tcpdump (against the loopback interface (lo) in this case) for finding the cause, as neither the ESB logs (even with SSL debug logs enabled: -Djavax.net.debug=all) nor the vsftpd log would show the underlying cause properly:

# tcpdump -A -vvv -i lo
...
06:41:37.566325 IP (tos 0x0, ttl 64, id 14724, offset 0, flags [DF], proto TCP (6), length 83)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe47 (incorrect -> 0xc3fd), seq 21:52, ack 11, win 342, options [nop,nop,TS val 37556708 ecr 37556708], length 31: FTP, length: 31
	234 Proceed with negotiation.
E..S9.@.@..............t..(u..,....V.G.....
.=...=..234 Proceed with negotiation.

06:41:37.576134 IP (tos 0x0, ttl 64, id 15677, offset 0, flags [DF], proto TCP (6), length 294)
    localhost.60532 > localhost.ftp: Flags [P.], cksum 0xff1a (incorrect -> 0x9887), seq 11:253, ack 52, win 342, options [nop,nop,TS val 37556711 ecr 37556708], length 242: FTP, length: 242
E..&==@.@............t....,...(....V.......
.=...=.............W*...l....I...TG	R.1_........F>...d.$.(.=.&.*.k.j.
...5.....9.8.#.'.<.%.).g.@.	.../.....3.2.,.+.0.....2...../...-.1.........
.............\.
.4.2...............	.
....................................................................
06:41:37.576284 IP (tos 0x0, ttl 64, id 14725, offset 0, flags [DF], proto TCP (6), length 59)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe2f (incorrect -> 0xd702), seq 52:59, ack 253, win 350, options [nop,nop,TS val 37556711 ecr 37556711], length 7: FTP, length: 7
E..;9.@.@..6...........t..(...-....^./.....
.=...=........(
06:41:37.577983 IP (tos 0x0, ttl 64, id 14726, offset 0, flags [DF], proto TCP (6), length 62)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe32 (incorrect -> 0xd9ea), seq 59:69, ack 253, win 350, options [nop,nop,TS val 37556711 ecr 37556711], length 10: FTP, length: 10
	500 OOPS: [!ftp]
E..>9.@.@..2...........t..(...-....^.2.....
.=...=..500 OOPS: 
06:41:37.577995 IP (tos 0x0, ttl 64, id 14727, offset 0, flags [DF], proto TCP (6), length 118)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe6a (incorrect -> 0xe009), seq 69:135, ack 253, win 350, options [nop,nop,TS val 37556711 ecr 37556711], length 66: FTP, length: 66
	error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher[!ftp]
E..v9.@.@..............t..(...-....^.j.....
.=...=..error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher
06:41:37.578000 IP (tos 0x0, ttl 64, id 14728, offset 0, flags [DF], proto TCP (6), length 54)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe2a (incorrect -> 0x0bb0), seq 135:137, ack 253, win 350, options [nop,nop,TS val 37556711 ecr 37556711], length 2: FTP, length: 2
	
E..69.@.@..8...........t..(...-....^.*.....
.=...=..

06:41:37.578230 IP (tos 0x0, ttl 64, id 14729, offset 0, flags [DF], proto TCP (6), length 62)
    localhost.ftp > localhost.60532: Flags [P.], cksum 0xfe32 (incorrect -> 0xd99c), seq 137:147, ack 253, win 350, options [nop,nop,TS val 37556711 ecr 37556711], length 10: FTP, length: 10
	500 OOPS: [!ftp]
E..>9.@.@../...........t..(...-....^.2.....
.=...=..500 OOPS: 
06:41:37.578323 IP (tos 0x0, ttl 64, id 15678, offset 0, flags [DF], proto TCP (6), length 52)
    localhost.60532 > localhost.ftp: Flags [R.], cksum 0xfe28 (incorrect -> 0x18bc), seq 253, ack 147, win 342, options [nop,nop,TS val 37556711 ecr 37556711], length 0
E..4=>@.@............t....-...(....V.(.....
.=...=..

Now we have a hint on what might be going wrong:

error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher[!ftp]

This usually occurs if the cipher suite offered by the FTP/S server is not sufficiently strong. Adding/updating

ssl_ciphers=HIGH

in vsftpd.conf would get you through this issue.

Once you have everything in place, the UESB trace would indicate a successful transfer:

2016-05-04 23:02:08,930 [-] [primary-1] [system] [000000I] DEBUG FTPSConnector Connecting over FTPS to URL : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no
220 (vsFTPd 3.0.3)
AUTH TLS
234 Proceed with negotiation.
PROT P
200 PROT now Private.
USER ftp
331 Please specify the password.
PASS ftp
230 Login successful.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (127,0,0,1,104,213).
STOR /srv/ftp/ftps/send.msg
150 Ok to send data.
226 Transfer complete.
2016-05-04 23:02:09,360 [-] [primary-1] [system] [000000I] DEBUG FTPConnector FTP copy completed successfully to : ftps://ftp:********@localhost/srv/ftp/ftps/send.msg?StrictHostKeyChecking=no

No comments:

Post a Comment