Proposal: Virtualization of Filesystem Operations for Lua 5.2.

[Georg Lehner <jorge-lua@...>]

Hello!

I propose to include a virtualization layer for filesystem operations in 
Lua 5.2.

Problems with the current approach to filesystem operations are:

- incomplete functionality (e.g. directory operations)
- their behaviour depends on the platform where Lua is built
- in some environments (deeply embedded..) the current functionality 
might be
  meaningless (and must be stripped out when porting)

Advantages of virtualization, besides releaving the problems, are (see 
also [1])

 - improved portability both for the Lua interpreter and for Lua scripts
 - extensibility, generalization and reusability:  Write portable Lua 
scripts
   for anything what can be made to look like a filesystem.

Tcl/Tk features a successful filesystem virtualization layer[2].

A simplified proof of concept implementation is available at [1] with 
about 400 lines
of Lua code.  I claim that an implementation of the virtualization layer 
will be
slightly less code then actually contained in the Lua interpreter 
(without including
a "real" filesystem driver).


I propose the following features (in more or less erratic order):

Extension of existing functions

- open(path,"d+") creates a directory
- open(path,"d"[,attributes]) returns an iterator over the directory path,
  returning filenames of existing files matching the attributes table
- remove(path) also removes (empty) directories


Additional functionality

- stat(path,[attributes]) sets/gets file attributes of path

- cd([path]) sets/gets the working directory of the current function
- mount(volume, path[,options..]) attaches the volume (see later) onto
  the mount point path
- bind(from, to [options..]) creates links between directory trees
- unmount(volume[, path]) deattaches volumes or links from the mount point

See the paper "Lexical File Names in Plan 9 or Getting Dot-Dot Right" from
Rob Pike [3] for the proposed mount/bind/unmount semantic

Implementation specifics

- each function has its own filesystem, which is by default inherited from
  its creator, but can be restricted (for sandboxes)
- '/' is the path separator, each filesystem driver provides translation to
  its specific path representation

- filesystem drivers implement only low level access to filesystem objects:
  read/write strings from/to files, create/delete files and directories,
  query directories and file attributes.
- the virtualization layer implements higher level features like lines(),
  formatted read and write.
- Lua scripts and pure Lua libraries implement high level applications like
  file copying or removing.

- A lua state has to be set up by its creator (e.g. the standalone Lua
  interpreter) to mount a specific filesystem driver.  This is however an
  optional step
- The current functionality of the io and os library can be maintained in
  the Lua interpreter and expanded as a filesystem driver which is mounted
  by default.  This maintains backwards compatiblity for existing Lua 
script
  without to much hassle and overhead.

- Volume instance specific identifiers like "C:", "D:",
  "http://www.magma.com.ni/sw/lua/luafs"; are set up as the "root" of a
  volume by the file system driver.


Permissions and Authorization

- Authorization can be done when a volume is instantiated, i.e. not Lua
  interpreter specific but filesystem driver specific
- Lua only needs to operate on already accessible filesystem objects, 
permission
  manipulation can be filessystem driver specific


Terms

- filesystem object: any object which can be represented by a directed graph
  with named edges and where each node can hold a set of named objects 
with
  a byte stream content and optional attributes
- filesystem driver: Lua library which operates on a specific class of
  filesystem objects
- volume: filesystem driver instance which operates on a filesystem object.


Filesystem class examples

- Lua tables
- Windows Registry
- Websites
- IMAP Mailaccounts
- Floppies, USB-Sticks, lokal and remote Windows drives
- zipfiles, tarballs, any other archives
- source repositories, git, svn, ...
- C-String arrays/data structures for deeply embedded Lua


The case of the Lua module search path

- with the proposed union mounts, search paths for Lua modules
  might become obsoleted by binding all locations where required
  modules live to a standard path in the virtual filesystems namespace

Best Regards,

    Jorge-León


[1] http://sw.magma.com.ni/LuaFs
[2] http://wiki.tcl.tk/2138
[3] http://doc.cat-v.org/plan_9/4th_edition/papers/lexnames