Using pyinotify with eventlet

For Logshipper I needed to monitor directories for modifications. On Linux, this can be achieved using INotify. In typical python fashion, there already is a package which provides inotify in python, aptly named pyinotify. While the package works reasonably well, its code isn’t in great shape, lacking tests, public CI and active maintenance. I’m also not a fan of the split code base for python 2 and python 3. Nevertheless it is a useful package which works as advertised. pyinotify works by creating a special file descriptor, and using select.poll to wait for it to be readable.

eventlet is a Python library for asynchronous IO. It allows you to easily write code that parallelizes any IO. Even though its website claims eventlet is intended for concurrent networking, it parallelizes file IO just as well. It works by patching various python modules, to call some global event loop whenever some IO is performed. Unfortunately, select.poll is not among the patched objects.

To use pyinotify in an eventlet driven application, I had to write a notifier for it. In pyinotify, the main function of a notifier is to wait for the file descriptor to become ready for reading. As eventlet is well equipped to do that job, this is quite a simple thing to do.

To adapt a pyinotify.Notifer, we simply need to derive from it. All of the hard work of interpreting the events and dispatching them is then left to pyinotify‘s implementation. In the constructor, we clean up the select.poll object during construction, as we won’t be using it.

1
2
3
4
5
6
7
8

class Notifier(pyinotify.Notifier):

    def __init__(self, *pargs, **kwargs):
        pyinotify.Notifier.__init__(self, *pargs, **kwargs)

        # We won't be using the pollobj
        self._pollobj.unregister(self._fd)
        self._pollobj = None

The original implementation cleans up the select.poll object in the stop method, so we’ll have to override that. We still need to close the inotify file descriptor, though.

1
2

    def stop(self):
        os.close(self._fd)

Then for the heart of this notifier, we call select.select, which can be patched by eventlet. Inotify is read-only, so we only need to select for read-ready file descriptors. There is some inconsistency between python 2 and python 3 as to what select.error looks like, so we use six to distinguish between the two.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

    def check_events(self, timeout=None):
        while True:
            try:
                read_fs, write_fs, x_fs = select.select([self._fd], [], [])
                break
            except select.error as err:
                error_no = (err[0] if six.PY2 else err.errno)
                if error_no == errno.EINTR:
                    break
                else:
                    raise

        return bool(read_fs)

While this implementation isn’t perfect, it does the job, which is what counts in the end. The original implementation had a timeout parameter, which now gets ignored, but for me that’s not an issue.