Virtual File System

The "Virtual File System" (VFS) was designed by Matt Newman and later adopted for use in Tclkit to greatly simplify and en The ZIP archive driver has already been described. It provides transparent access to the de-facto ZIP archive format. To use it, do "package require vfs.zip".

The Metakit driver provides access to scripted documents (package require mk4vfs). With this driver, you can store and access any number of "files" stored in the Metakit database. Compression can be used (zlib) as default by setting a global flag ($mk4vfs::compressed), or on a file-by-file basis by opening the file in mode "wb". Apart from a few ideosyncracies mentioned below, your scripts never need to know the difference between having data inside a single scripted document, or keeping them in more traditional files.

The "namespace driver" (package require vfs.ns) is an unusual one, which stretches the boundaries of what a "file system" is: when you mount its "vars" filesystem, you get access to all *variables* in the current process. Each namespace is a directory, each variable is a file. Reading such a file is equivalent to getting the contents of that variable, writing it *set* the variable. When you mount the "procs" filesystem, things become even weirder: each proc in the running process becomes a file, which you read *and* write. If this approach doesn't make sense to you, please just ignore it. It's an experimental feature, which you can safely ignore. The vfs.ns driver is an interesting example in its own right though, of how to write a VFS driver.

VFS is not perfect

Although VFS is suprisingly transparant when used with Tcl scripts, it does have a few limitations right now.

- The illusion of virtual files breaks down outside Tcl: though you can "cd" into a virtual directory, and "glob" it, you cannot expect "exec" to be aware of this. When running code which does not rely on the *Tcl* open/read/glob/etc. command, all the magic of VFS breaks down. There is very little one can do about this - VFS is a concepts which works within the boundaries of a Tcl/Tk process. Trying to improve this requires OS/kernel support, something which is way beyond the scope of VFS. Note that VFS is fully portable, it works equally well in Windows, Unix, and Macintosh. But only "inside" the Tcl application.

- There is a performance drop: VFS takes over almost all I/O operations (except the most important ones luckily: read & write) and re-routes these calls through Tcl to the appropriate driver. Even if you do not mount anything with VFS, just loading VFS will already have this effect, since open/glob/source/load and more are all replaced by Tcl-scripted equivalents. The original "real" commands will be moved to the tcl::* namespace.

- Paths tend to get longer, and unfortunately that slows down VFS even further, because it has to disect FS paths to figure out which driver to use. The most important bottlenecks have already been addressed in recent (Tcl 8.2/8.3) releases, the remaining one is taken care of by a C extension which is part of Tclkit, called "vfs::normalize" (package vfs.builtin).

- As of Tcl 8.3.2, the "info" command no longer needs to be overriden by VFS, which is a big advantage since "info exists" now remains at C speed. The culprit was "info source", which has to be modified for VFS use but which now accepts an extra argument to do this properly.

- Things like "source" work fine in VFS (it is one of the overriden commands), meaning that you can source a script from any VFS-mounted "file".

- Even "load sharedlib" works, because VFS implements a powerful replacement of it. What it does, when the shared library comes from a VFS file, is to copy that file to a temporary one in the real file system, and then load that copy instead. This is suprisingly effective *most* of the time, but not always. If the loaded library needs other libraries, it might not find those. Note that the copy is placed in a special place, on Unix it would be "/tmp/tclkit/..." on other systems that will vary. Things like the TMP/TEMP environment variable are checked to find a suitable place. The copied file is *not* cleaned up. This speeds up subsequent runs, but also leaves some "debris" on the disk...

- Encodings don't work (fixed in 8.4-20). They use Tcl_OpenFileChannel, which is a C-level call, and hence not VFS-aware. This is he same reason as why the Tk "image" command doesn't open files inside VFS.

- VFS is not clever enough to detect command "short-forms", i.e. "file isdir" will not work, use "file isdirectory" instead.

A few nasty gotcha's

- Tk's image command is one of the cases where VFS does not yet work properly (fixed in 8.4-20). Image can take a "-file" argument, but being implemented in the C core, it has no idea that VFS might be trying to fake things. The workaround is to load a script called "compat.tcl", written by Steve Landers, which replaces the image command by one which looks for such a "-file" argument, loads the data through VFS, and then passes it to the real image command as "-data".

- A similar case occurs in the WinIco extension. The proper solution would be to virtualize file system I/O at a deeper level, i.e. change all current C code in the core to no use "fopen" calls, but to set up a Tcl channel and use that for I/O instead (done, a fixed version is in 8.4-20).

- Files opened for writing or appending need to be closed to end up properly inside a VFS filesystem, such as mk4vfs. The reason for this is that at exit, files do get closed automatically by Tcl, but the order in which this happens might be too late - the VFS database file might already have been cleaned up, for example. The safe thing to do is to close files. Note that flushing a file is not always enough in the case of mk4vfs, because when compression is used only the final close will actually store the last bits of data. ---- blah blah fuckitydoodah Nov 3, 2000 - The problem with C-level code opening files has been resolved, as far as Tcl channels are concerned (which is basically everything in Tcl itself). The new Tcl_OpenFileChannel intercept logic in Tcl 8.4a2 (has it been around much longer?) is now used if available. That means "image" works again, and so do bitmaps, and all other parts of Tcl doing file I/O. Including Unicode encodings, which are functional as of the new Tclkit-8.4-20 release.

Nov 8, 2001 - Er, yes. In the latest prebuilt Tclkit ( https://www.equi4.com/pub/tk/tclkit-8.4-26-linux/ ) encodings most definitely don't work. -EE