Re: Proposal: Virtualization of Filesystem Operations for Lua 5.2.

["Timm S. Mueller" <tmueller@...>]

On Wed, 02 Nov 2011 20:53:24 +0100
Georg Lehner <jorge-lua@magma.com.ni> wrote:

Hello Georg,
 
> I propose to include a virtualization layer for filesystem operations in 
> Lua 5.2.

> [..]

not to criticize your proposal, just a few general thoughts, as I was
once concerned with the same issue.

I would come out in opposition to any half-baked solution becoming part
of the Lua core. Everybody is free to implement an additional
filesystem library of her own, everybody is free to modify the core,
and everybody is free to link Lua against a different C library, which
may implement nifty features.

Modifications to the Lua core should not impair its simplicity,
transparency, and to some extent, its size and performance. The sooner
you call out to a library, the better. Sticking to what is offered
through stdio.h is one of the greater achievements of Lua. Any
complication can be justified only if it offers a comprehensive
solution to real problems.

I have once written a vfs in the shape of a filesystem and device
driver architecture for some kind of embedded operating system. It
supported mounting, multiple roots, loops, asynchronous I/O, locking,
automounting etc. Lua was adapted to this and part of the distribution.
The result was fully transparent, in that require, dofile, module
loading, and even stdin and stdout went through the vfs layer, and
modules could be loaded from mountable loop devices such as zip files.
Stdio was shielded completely, not just a little here and there. [1]

But while this is what I would call a comprehensible solution to a real
problem, I would recommend against it in the general case:

1. The vfs was too large and complex.

2. The vfs layer was written in C and had no knowledge of Lua, hence
new drivers could only be written in C. From a Lua perspective, I would
find this unacceptable.

3. The vfs introduced concepts which may look unfamiliar to people
programming applications on Linux or Windows (directory handles, file
handles and filesystem locks all based on the same 'IO packet', for
example, which could be used for asynchronous I/O directly).

4. The vfs introduced a distinct filesystem notation of its own, which
may look even more unfamiliar - it did not resemble either Windows or
Unix. I suspect that this would be hardly tolerable for the general
purpose.

On the other hand, I would just love to have some 'hooks' in the Lua
core to plug in a filesystem layer - but not just one, ANY filesystem
layer. For the chicken-egg-condition that it imposes, it may appear as
if a vfs should be linked in statically and become part of the Lua
core. But I would suggest to think again and to come up with just the
hooks that allow for dynamic reprogramming of a Lua filesystem's
semantics, i.e. a fully require'able vfs that, once it is loaded, is
also retargeted to by the 'io' library. (New functions, OTOH, should
probably go into a library table of its own.)

It would be very gratifying if someone had a go at it and tried to come
up with such a comprehensible vfs design in mind (no need to actually
implement it right at the outset), and just the (hopefully few,
surgical) hooks needed in the Lua core. Then I'm all for it.

- Timm

[1] http://hg.neoscientists.org/teklib/file/418c081f184c/doc/text/io.txt

-- 
Timm S. Mueller <tmueller@schulze-mueller.de>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/