Wednesday, June 4, 2008

Cross-Platform Monitoring of Filesystem Events

A recent problem with deployment of the Ringlight client has been that users with a large number of folders have been experiencing annoying amounts of CPU usage. This is because the most fundamental functionality that Ringlight provides is periodically rescanning your filesystem to automatically find changes to the filesystem. Rescan too infrequently and changes won't appear on the website when expected, confusing users. Rescan too frequently and users will complain about too much CPU usage. There are many applications that require rescanning the filesystem, from virus scanners to automatic backup programs, and they all have to deal with this problem.

An attractive alternative to rescanning the filesystem is to use filesystem monitoring events. Rather than periodically scanning to see if anything has changed, instead let the operating system notify you when something has change. Very efficient! Unfortunately, unlike a simple recursive traversal of directories, this approach has to be implemented separately on each major platform and each OS has its own pitfalls and gotchas. I will focus primarily on building this in python, although the same underlying mechanisms can be used in any language with appropriate bindings.

On Windows, filesystem events are available using the Python for Windows Extensions. It is not a particularly simple API to use, being a direct binding to the Windows system calls.

On OS X, PyKQueue offers a binding to kqueue, which is also available on BSD.

On Linux, there are two kernel interfaces, dnotify and inotify, depending on whether the kernel version is lesser or greater than 2.6.13. You can call inotify directly with pyinotify. You can also use a more generic library such as Gamin, which will use either inotify or dnotify, whichever is available. Of course, really old versions of Linux don't even have dnotify, and you'll have to fall back to periodic rescanning.

Every platform's filesystem monitoring API is different and each has different issues, however they generally share a common set of issues as well:

  • Network drives don't generate events - you'll have to use periodic scanning for these
  • Every folder to be watched must be registered separately - you can't request notifications for an entire directory tree. You can to register all the directories separately and when a new directory is create you have to remember to register it as well. You sometimes need to keep a file descriptor around for each directory you're watching, so watch out for running out of file descriptors.
  • No special handling is done for shortcuts, aliases, or symlinks - if you're monitoring, say, a directory, and that directory has a shortcut (or the equivalent for that OS), you need to monitor two objects: the shortcut itself (in case its target is changed), and the targeted file or directory.
  • Sometimes deleting a directory won't send deletion events for files in that directory or subdirectories. You have to maintain a copy of this information yourself and perform a virtual recursive delete on your database when the parent directory deletion event is received.
The Ringlight client, being a cross-platform application the primary function of which is to monitor filesystem changes (and report them to the server, where the real fun happens), naturally takes all this into account. I am planning on release the filesystem monitoring core of the Ringlight client as open source, as there are no good cross-platform filesystem monitoring libraries available and it's really a shame that people have to reimplement all of this for their applications.

By the way, the users seem quite happy with the new version of the client that users filesystem monitoring events. No complaints about excessive CPU usage anymore!


Jim said...

You are quite incorrect about several of your claims regarding problems with filesystem monitoring. Having written a couple of such cross-platform tools in Python over the years you apparently need to do a bit more digging.

Since it is the dominant platform at the moment I will outline some of the issues you need to deal with on win32. For starters, check up on Tim Golden's example wrappers for ReadDirectoryChangesW(). This is what you will need to use for windows to grab filesystem changes. And yes, you can monitor an entire disk using a single call to ReadDirectoryChangesW (which eliminates your problems regarding symlinks/aliases.)

What you will need to modify in the example code is to change the call to ReadDirectoryChangesW to one that uses an io completion port so that your watcher thread is not blocked waiting for a filesystem event to occur (this will cause some of the watchers on filesystems that do not change much to be hard to kill, as you will have to wait for an event to fire before the call will return.) Once you have the wrappers tweaked so that you can kill watcher threads at will you will need to add support to detect insertion and removal request for usb devices. You will need to add a message hook for WM_DEVICECHANGE notification so that when the user requests usb device removal you can kill your watcher thread and then pass the device change notice along. Without this you will effectively lock a usb device as soon as you put a watcher on it because the request for device removal will be killed by the open filehandle you have in the ReadDirectoryChangesW call. What gets tricky about catching the WM_DEVICECHANGE hook is that you have to run this particular windows message pump in the main thread of your application (and run all other event loops in sub-threads) or else windows will never hand it the device change notifications.

It is ugly all the way down here, but once you get the basics working it is really easy to have a large batch of threads watching entire drives and throwing the events they catch into a queue for another thread to process. Matching rename_from and rename_to messages it a bit of a PITA, but not too painful considering all of the other contortions necessary to get that basic filesystem watcher code working.

The various threads to do the filesystem watching consume very little CPU and once you have a good filesystem watcher working you will need to rescan the disk a lot less frequently.

blanu said...

Hi Jim, thanks for taking the time to leave a comment.

I do in fact use ReadDirectoryChangeW() on Windows, which is part of the Python for Windows Extensions that I mention in the article. You are correct that my bullet point regarding a lack of recursive monitoring doesn't hold on Windows. (Thanks, Windows!)

However, the problem with symlinks (or in the case of Windows, shortcuts) does apply to Windows. If you register a directory tree and the directory contains a shortcut to a subdirectory to somewhere else (not contained in the directory tree), you will not receive events when the contents of the subdirectory change. I understand that your point is that one approach is to just monitor the entire filesystem. So this particular point is of more importance if you are only monitoring part of the filesystem (for instance, one directory, perhaps a "hot folder"). While you could achieve this by monitoring the whole filesystem, and simply discarding events not relating to the hot folder, you do need to do some extra bookkeeping to match events on shortcut destination folders as relevant.

Thanks for the tip regarding supporting USB storage devices properly on Windows. I had not thought of that, and it is very useful!

Anonymous said...

"I am planning on release the filesystem monitoring core of the Ringlight client as open source, as there are no good cross-platform filesystem monitoring libraries available and it's really a shame that people have to reimplement all of this for their applications."

That is somthing I would love to see. It is very interesting, and yes, there is very little out there. Any chance of a sneak peak?


blanu said...

Ringlight is launching a public beta this month. It will take some time after that to package the filesystem monitoring component into a usable library. I'll keep you updated!

Anonymous said...

MMh. On OS X you also don't have to watch every directory:
FSEvents.h doesn't have the kqueue's problem.

Andrew Pendleton said...

inotify, using pyinotify, can also do recursive watches, though it's not on by default. You just have to pass rec=True to the add_watch function of the watch manager (also, probably auto_add=True, if you want newly created subdirectories to be automatically monitored).

In any event, I'm working on some code that, for the moment, depends on pyinotify and is thus Linux-specific, but that I wouldn't mind making cross-platform, so I'm very much interested in your progress.

swaits said...

Did you ever release this code?

phobis said...

Is there any update on the code being published for this? Thanks :)

Anonymous said...

Me too. I would like to play with the source as well.

Gora said...

I'm working on a Python library that attempts to monitor file system events on operating systems and uses the native APIs as much as possible before resorting to polling the disk as a fallback.

Check it out here:

Simon said...

I know its been a while, but I'm really interested in your cross-platform code for my own open source project. Even if its raw cr*p code, it would be helpful. Can you post it?