The Java NIO WatchService: Pitfalls to watchOut() for

My intention here is not to delve into the API or whereabouts of the Java NIO watch service; in a nutshell, it allows an application to register for filesystem event notifications on a set of arbitrary paths. This is quite useful in text editors, filesystem monitors and other applications which require immediate notifications without incurring the overhead of repetitively polling the file tree. You can find perfect explanations of it on the internet, such as this article, accompanied by a plethora of examples and tutorials.

My article is dedicated solely for explaining the various pitfalls I came across when writing my first WatchService-based application—an event-driven file transport for the UltraESB, the best open source ESB ever.

If you are familiar with inotify watches on Linux, you won't need any further introduction to NIO watch API. Although I have not directly used inotify, judging by the content of inotify's man page I would say that the NIO watch service, on Linux, is just a simplified wrapper module for inotify. The architecture is quite identical, with the WatchService corresponding to the inotify instance, the WatchKey corresponding to the watch descriptor and the WatchEvent corresponding to the inotify_event entity; and so are the processes of watch registration and retrieval of events (blocking and non-blocking). NIO seems to have introduced the concept of queueing events under WatchKey's and requiring the user to reset() the WatchKey before it can gather further events, but almost everything else is quite the same as inotify.

First, it should be noted that in some systems (such as Mac OS X) the watch service may fall back internally to a polling approach, if notifications are not natively supported by the underlying OS. It is also known that the watch service has certain incompatibilities with Windows-based operating systems. From what I have read so far, *NIX environments are the ones which can reap the true benefits of the watch service.

But that's just the beginning.

This extract from the inotify manual itself, titled Limitations and caveats, gives us an idea of how careful we should be, when using inotify-based implementations:

---

Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for large directory trees.

The inotify API provides no information about the user or process that triggered the inotify event. In particular, there is no easy way for a process that is monitoring events via inotify to distinguish events that it triggers itself from those that are triggered by other processes.

Note that the event queue can overflow. In this case, events are lost. Robust applications should handle the possibility of lost events gracefully.

The inotify API identifies affected files by filename. However, by the time an application processes an inotify event, the filename may already have been deleted or renamed.

If monitoring an entire directory subtree, and a new subdirectory is created in that tree, be aware that by the time you create a watch for the new subdirectory, new files may already have been created in the subdirectory. Therefore, you may want to scan the contents of the subdirectory immediately after adding the watch.

---

On most systems, the number of WatchService instances you can open is limited. Exceeding this limit can sometimes produce vague errors like "network BIOS command limit reached". However, you'll almost always work with a single WatchService per application, so this should rarely become an actual problem. Nevertheless, keep in mind the fact that the WatchService is another I/O resource (just like a Stream or Channel) and make sure that you invoke close() on it after use, to avoid resource exhaustion.

In addition to the basic watch event types (ENTRY_CREATE, ENTRY_MODIFY and ENTRY_DELETE), there's a fourth event type, OVERFLOW. This corresponds to the case where the event queue for a given WatchKey has overflown, causing the loss of events as explained by inotify. For example, under default Linux settings, a WatchKey will produce an OVERFLOW if its queued event count exceeds 512. Although this limit is configurable, it should be remembered that an overflow is not entirely unavoidable, and should be taken into considerable in all mission-critical implementations.

The Path (more correctly, Watchable instance) associated with a WatchKey can be retrieved via its watchable() method. However, the watchable() is bound to the system representation of the directory (e.g. inode on Linux), and not the actual (literal/textual) path. Hence, even if we change the name of a directory in the path, the value of watchable() would remain at the old value, eventually leading to failure when we try to use watchable()'s value as a valid Path. Hence it is necessary to constantly keep track of changes in directory paths and update the local 'image' of the filesystem accordingly. Fortunately, a folder rename or move (that would result in the invalidation of a corresponding watchable() value) is always accompanied by an ENTRY_DELETE event for that directory, so we can quickly identify and take action before things get messed up.

Events are not guaranteed to arrive in any particular order. For example, if you delete a directory containing a subdirectory and being watched by watch service W, located inside a directory which is also being watched by W (so there are is a 3-level directory hierarchy), deletion of the middle directory may be notified before that of the inner directory, although logically they should have happened in the opposite order. To make things worse, if the innermost directory was also being watched, it might generate a bogus notification (i.e. one containing zero events) during deletion.

Every operation is decomposed into the 3 basic event types. For example, renaming a file or directory inside a watched parent directory will trigger an ENTRY_DELETE (oops, that file is gone!) followed by an ENTRY_CREATE (hey, a new file is here!). This can be thought of as a new way of looking at the renaming process (deletion followed by creation), but it does not reflect the actual operation (inode metadata update) that would be happening behind the scenes.

A more strange (albeit perfectly OK) scenario happens if a file is moved into or out of a watched directory. As only the outermost directory entry gets changed 'effectively', you only get a notification for the outermost directory that got moved. Now, if there were any watched subdirectories inside the moved directory, they would still be 'active' but their watchable()s would no longer be valid; from that point onwards, only the relative path provided by the corresponding watch key would be valid, and you will have to manually prepend it with the new parent path to get the correct full path.

Enough talking; let's demonstrate all this Greek with a simple directory hierarchy!

a
+--b
|  +--d
|  |  +--g.txt
|  |
|  +--e
|     +--h.log
+--c
   +--f
      +--i.sh

Assume that all directories except f are being watched. See if you can deduce what is going on behind the scenes.

Operation	Resulting events
	WatchKey's watchable()	WatchEvent's kind()	WatchEvent's context()
delete c	a	ENTRY_DELETE	c
	c	ENTRY_DELETE	f
	(f deleted silently; order of other events is unpredictable)
move c into b	a	ENTRY_DELETE	c
	b	ENTRY_CREATE	c
move e into f (not included here), then delete f	c	ENTRY_DELETE	f
	e	ENTRY_DELETE	h.log
	(f deleted silently; event order unpredictable)
rename c as j	a	ENTRY_DELETE	c
	a	ENTRY_CREATE	j
delete d	d	ENTRY_DELETE	g.txt
	b	ENTRY_DELETE	d

Moral of the story? Be careful and thoughtful when using the WatchService API. It's a golden sword that can cut you badly if handled in the wrong way.