Google's Summer of Code: Part IV

NetBSD-SoC: Efficient Memory Filesystem

NetBSD inherited the Memory File System (MFS), a filesystem that uses virtual memory pages to store its contents, from 4.4BSD. MFS implements the on-disk Fast File System-s lower layer to store blocks into memory. This is inefficient because the same algorithms used to lay out data on disk are used for memory, introducing redundancy and complex data treatment. Despite that, the worst problem with MFS is that, once it has mapped a page, it never releases it.

tmpfs (short for "temporary filesystem") solves these problems. It is a completely new filesystem that works with NetBSD-s virtual memory subsystem, UVM. It aims to be fast and use memory efficiently, which includes releasing it as soon as possible (for instance, as a result of a file deletion).

At this writing, tmpfs is already integrated into NetBSD-current (3.99.8), which makes testing and further development a lot easier. All the filesystem's functionality is written and works, albeit with some known bugs. The most significant problem is that it currently uses wired (nonswappable) kernel memory to store file metadata.

Let's now dive a bit into tmpfs' internals (full documentation is included in the project's tmpfs(9) manual page). As with traditional filesystems, tmpfs organizes all the files stored within it in a tree graph. Each file has a node associated with it, which describes its attributes (owner, group, mode, and the like) and all the type-specific information. For example, a character device has a major and minor number, a symbolic link has the name of its target, and so on.

Directories are implemented using linked lists, making their management extremely easy. Directory traversal is quick, and node creation and deletion need not be concerned with entry proximity nor gaps between entries.

Regular files, on the other hand, are interesting because they are managed almost completely by the UVM. Each file has an anonymous memory object attached to it, encapsulating the memory allocated to the file. This memory is managed solely by the virtual memory subsystem. This design meshes well with the Unified Buffer Cache used by NetBSD, where files are UVM pagers, and file reading and writing is performed by mapping and accessing pages of the file-s UVM object. Thus, the only difference between files on other filesystems and those on tmpfs is the nature of the UVM object holding the pages.

Back to Top

Pyasynchio: The Python Asynchronous I/O Library

Name: Vladimir Sukhoy
Contact: [email protected]
School: Iowa State University
Major: Ph.D. candidate, Applied Mathematics
Project: Pyasynchio
Project Page: http://pyasynchio.berlios.de/pyasynchio-about.htm
Mentor: Mark Hammond
Mentoring Organization: The Python Software Foundation (http://www.python.org/)

Pyasynchio is a Python library created to support cross-platform asynchronous I/O (AIO) operations. Compared to the famous asyncore library, for example, pyasynchio is simple and does not require you to use object-oriented approaches. It just provides a way to initiate AIO and poll completion status. It is up to library users to implement patterns for processing operation results. At the moment, Pyasynchio supports file and socket AIO (only on Windows, but POSIX AIO and Solaris AIO support is coming). Pyasynchio uses built-in Python file and socket handles as AIO targets, so users can plug Pyasynchio into the middle of their code.

AIO is a way to alleviate I/O bottlenecks without multithreading. Usually, AIO is efficient in terms of resource usage (sometimes more efficient than multithreading, select/poll, or mixed solutions). AIO allows additional speed optimization at the OS and driver level and does not require an operation to complete during the call to the function that started it. A well-known object-oriented implementation of AIO on top of different low-level OS facilities can be seen in the ACE library (http://www.cs.wustl.edu/~schmidt/ACE.html).

It is important to note that AIO is different from synchronous nonblocking I/O (often, nonblocking I/O is erroneously called "asynchronous" I/O, but it really isn-t because each nonblocking I/O operation is still synchronous). Pyasynchio makes use of the Proactor pattern and Asynchronous Completion Token pattern, but the dynamic type system of Python makes some of Proactor-s object-oriented burden unnecessary. Applications of Pyasynchio may be even greater when Python introduces coroutines support (PEP 342). With coroutines, some I/O processing tasks can be coded in a synchronous manner (without distributing connectionprocessing logic over numerous handler classes/callbacks) but will still be able to take advantage of nonblocking I/O or AIO.

It is worth mentioning that a coroutine-based approach is possible with ordinary nonblocking synchronous I/O (as outlined in PEP 342), but AIO coroutines may be more effective in terms of resource usage and/or speed and/or implementation clarity. Pyasynchio code and binaries are available under an MIT license.

DDJ