Comments on Step Three: Profit!: Cross-Platform Monitoring of Filesystem Events

I know its been a while, but I'm really intere...

2011-06-16T16:22:31.233-07:00

I know its been a while, but I'm really interested in your cross-platform code for my own open source project. Even if its raw cr*p code, it would be helpful. Can you post it?

I'm working on a Python library that attempts ...

2010-11-19T02:45:58.822-08:00

I'm working on a Python library that attempts to monitor file system events on operating systems and uses the native APIs as much as possible before resorting to polling the disk as a fallback.

Check it out here:
http://github.com/gorakhargosh/watchdog/

Me too. I would like to play with the source as we...

2009-08-26T14:09:07.374-07:00

Me too. I would like to play with the source as well.

Is there any update on the code being published fo...

2009-06-19T13:10:24.734-07:00

Is there any update on the code being published for this? Thanks :)

Did you ever release this code?

2009-01-07T15:06:00.000-08:00

Did you ever release this code?

inotify, using pyinotify, can also do recursive wa...

2008-12-10T18:24:00.000-08:00

inotify, using pyinotify, can also do recursive watches, though it's not on by default. You just have to pass rec=True to the add_watch function of the watch manager (also, probably auto_add=True, if you want newly created subdirectories to be automatically monitored).

In any event, I'm working on some code that, for the moment, depends on pyinotify and is thus Linux-specific, but that I wouldn't mind making cross-platform, so I'm very much interested in your progress.

MMh. On OS X you also don't have to watch every di...

2008-11-07T02:10:00.000-08:00

MMh. On OS X you also don't have to watch every directory:
FSEvents.h doesn't have the kqueue's problem.

Ringlight is launching a public beta this month. I...

2008-10-03T09:26:00.000-07:00

Ringlight is launching a public beta this month. It will take some time after that to package the filesystem monitoring component into a usable library. I'll keep you updated!

"I am planning on release the filesystem monitorin...

2008-10-01T19:19:00.000-07:00

"I am planning on release the filesystem monitoring core of the Ringlight client as open source, as there are no good cross-platform filesystem monitoring libraries available and it's really a shame that people have to reimplement all of this for their applications."

That is somthing I would love to see. It is very interesting, and yes, there is very little out there. Any chance of a sneak peak?

Pete

Hi Jim, thanks for taking the time to leave a comm...

2008-06-10T12:57:00.000-07:00

Hi Jim, thanks for taking the time to leave a comment.

I do in fact use ReadDirectoryChangeW() on Windows, which is part of the Python for Windows Extensions that I mention in the article. You are correct that my bullet point regarding a lack of recursive monitoring doesn't hold on Windows. (Thanks, Windows!)

However, the problem with symlinks (or in the case of Windows, shortcuts) does apply to Windows. If you register a directory tree and the directory contains a shortcut to a subdirectory to somewhere else (not contained in the directory tree), you will not receive events when the contents of the subdirectory change. I understand that your point is that one approach is to just monitor the entire filesystem. So this particular point is of more importance if you are only monitoring part of the filesystem (for instance, one directory, perhaps a "hot folder"). While you could achieve this by monitoring the whole filesystem, and simply discarding events not relating to the hot folder, you do need to do some extra bookkeeping to match events on shortcut destination folders as relevant.

Thanks for the tip regarding supporting USB storage devices properly on Windows. I had not thought of that, and it is very useful!

You are quite incorrect about several of your clai...

2008-06-06T11:47:00.000-07:00

You are quite incorrect about several of your claims regarding problems with filesystem monitoring. Having written a couple of such cross-platform tools in Python over the years you apparently need to do a bit more digging.

Since it is the dominant platform at the moment I will outline some of the issues you need to deal with on win32. For starters, check up on Tim Golden's example wrappers for ReadDirectoryChangesW(). This is what you will need to use for windows to grab filesystem changes. And yes, you can monitor an entire disk using a single call to ReadDirectoryChangesW (which eliminates your problems regarding symlinks/aliases.)

What you will need to modify in the example code is to change the call to ReadDirectoryChangesW to one that uses an io completion port so that your watcher thread is not blocked waiting for a filesystem event to occur (this will cause some of the watchers on filesystems that do not change much to be hard to kill, as you will have to wait for an event to fire before the call will return.) Once you have the wrappers tweaked so that you can kill watcher threads at will you will need to add support to detect insertion and removal request for usb devices. You will need to add a message hook for WM_DEVICECHANGE notification so that when the user requests usb device removal you can kill your watcher thread and then pass the device change notice along. Without this you will effectively lock a usb device as soon as you put a watcher on it because the request for device removal will be killed by the open filehandle you have in the ReadDirectoryChangesW call. What gets tricky about catching the WM_DEVICECHANGE hook is that you have to run this particular windows message pump in the main thread of your application (and run all other event loops in sub-threads) or else windows will never hand it the device change notifications.

It is ugly all the way down here, but once you get the basics working it is really easy to have a large batch of threads watching entire drives and throwing the events they catch into a queue for another thread to process. Matching rename_from and rename_to messages it a bit of a PITA, but not too painful considering all of the other contortions necessary to get that basic filesystem watcher code working.

The various threads to do the filesystem watching consume very little CPU and once you have a good filesystem watcher working you will need to rescan the disk a lot less frequently.