Re: Proposal: Virtualization of Filesystem Operations for Lua 5.2.

[Sean Conner <sean@...>]

It was thus said that the Great Georg Lehner once stated:
> Hello!
> 
> Thanks for the interest and for all the inputs to the topic.
> 
> One thing I like about Lua is, that its simple and small.
> 
> The "fs" proposal tries to closely follow this principle. It's scope is 
> narrow:
> 
> - read/write contents of static files

  The calls in io handle this already, it's just that filenames are system
dependent.

> - manipulate and query filesystem trees

  
> This is enough to write a wealth of day-to-day utilities.  For the
> remaining part of the universe other solutions will do.
> 
> In my opinion, only the lowest common denominator of all (!) filesystems
> should be covered by a Lua core "fs", and this would be: (directory) trees
> with sets of named bytestreamcontainers (files) in each nodes.  Practical
> extensions:

  What's a "directory"?  Not all file systems have them.  Granted, they tend
to be nearly extinct nowadays, but it wouldn't surprise me to find some
obscure embedded system based off something like CP/M that had no concept of
directories (much like MS-DOS 1.0).  Is this something to be concerned
about?  Perhaps not.  

  But even so, you are still retricting yourself.  Back in college (for me,
this was the early 90s) I came up with a filesystem design that took the
whole "directory" concept to (what I like to think) mind-blowing
proportions.  A "directory" is little more than a file that maps names to
disk locations of other files.

	A directory is simply by adding a new type of file, the directory. A
	directory is accessed exactly as an ordinary file.  It contains
	16-byte entries consisting of a 14-byte name [1] and an i-number.

		"UNIX Implementation" by K. Thompson, The Bell System
		Technical Journal, July-August 1978, Vol. 57, No. 6, Part 2
		(I'll be quoting from this more below)

I took that to it's nearly literal conclusion and proposed a system whereby
you could "cd" into a structured file and "ls" components.  The primary
example I used at the time was an executable file, which contains a code
segment, a data segment, symbol segments, comment segments, etc.  The effect
was that the Unix command "strip" (which strips symbolic information from an
executable) simply became the "rm" command---something along the lines of:

	% rm build/program/symbols.segment

  With such a system, suddenly a lot of specialized tools suddenly go away
(say, code to edit comments in image files).  

> - directed graphs instead of trees

  Ah, Ken Thompson has something to say about that too, from his "UNIX
Implementation" paper:

	The file system structure allows an arbitrary, directed graph of
	directories with regular files linked in at artitrary places in this
	graph.  In fact, very early UNIX systems used such a structure. 
	Administration of such a structure became so chaotic that later
	systems were restricted to a directory tree.  Even now, with regular
	files linked multiply into artibrary places in the tree, accounting
	for space has become a problem.  It may become necessary to restrict
	the entire structure to a tree, and allow a new form of linking that
	is subservient to the tree structure.

  With a directed graph as a file system, you run the risk that memory
garbage collection systems have---that of self-referencing structures that
are never reclaimed.  It's seems like it's an easy way to hide files to me.

  With respect to the accounting issue, I just found a file that has nine
different names:

  -rw-r--r--  9 spc spc 95 Mar 25  2000 /home/spc/apps/djb/alloc/trycpp.c
              ^ 
              |
              +- there are nine names pointing to the data for this file

  Good luck in trying to find the other eight.

> - attributes for filesystem objects
> 
> While the unification of stream and block io in the unix world is a great
> thing, I tend to not make it part of the "fs" concept. stdin/stdout/stderr
> is a concept which might be available and supported on platforms of
> typical Lua standalone interpreters, but is not a feature found and needed
> in all situation where a Lua interpreter is embedded.  

  stdin, stdout and stderr are mandated by the ANSI C Standard.  Yes, there
are embedded platforms where the concept of stdin, stdout and stderr are
moot, but if you want to sell an ANSI-C compiler, you must provide an
implementation of the standard library as well.  But there's nothing saying
you have to *use* the standard library.

> Let the Lua
> embedding software do the respective work if required by the application. 
> Also: there is still "io" around to process stdin/stdout
> 
> 
> Some comments on the following two acpects which have been mentioned in
> the discussions:
> 
> UTF8 (or whatever other character set and format of) filenames:
> 
> "fs" is thought to be a namespace, where each path element is a Lua string
> (whatever a Lua string might be in some Lua incarnation) and the '/'
> character is the separator between paths and the symbol for the root of
> the namespace tree.

  Not every file system uses the '/' as a path separator.  Older verions of
Mac OS (pre-Unix) used ':' as a path separator.  

> Every subtree below a 'mount point' is handed over to
> the filesystem driver responsible for this exact 'mount point' (e.g.
> LuaFilesystem).  The filesystem drivers converts "fs" subtrees to-and-fro
> the underlying filesystem naming convention.

  So I want to create a file called "Supercalifragilisticexpialidocious: The
Musical" on a MS-DOS 12bFAT disk (8 alphanum characters for name, 3 for
extention).  Or a pre-Unix version of Mac OS.

  I think my question really is:  what exactly are you trying to solve?  If
it's "to make filesystem manipulations portable" then I refer you to the
solution Common Lisp did, and ask how many people like using that.

> Asynchronouos I/O:
> 
> This is a challenging topic and I would like to see a good proposal how to
> make it fit in nicely with the Lua core.
> 
> However I have the impression, that there is no standard way for
> concurrency in Lua except coroutines.  Adding "fileevents" or asynchronous
> I/O to the Lua core would require to add a central dispatching mechanism
> to it (event loop, wait queues, or the like) and this seems to much
> additional complexity and intrusion with respect to the scope of the "fs"
> proposal.

  It is.  My own play implementation of this is around 2,100 lines of C
code, and is restricted to Linux because of the use of epoll().  I myself
keep flipflopping on asynchronous IO, but in the long run, with large
multi-core system looming in the future, I'm guessing asynchronous IO will
loose out (because you can always using the easier to program synchronous IO
on another core and avoid the obfuscation of asynchronous IO, which is a
type of cooperative multitask anyway).

> - - -
> Both issues, however can be treated nicely *with* "fs", even if not *in* 
> "fs":
> 
> If a program wants to operate on file:
> 
> local path = "/this/is/a/lengthy/path/with/a/mountpount/and/file/below"
> 
> and e.g. the host root directory is mounted via a filesystem driver (io,
> os and LuaFilesystem) at "mountpoint" it will
> 
>   local fh = fs.open(path)
> 
> which will internally assign
> 
>   fh.fd = io.open("/and/file/below")
> 
> which stores the returned "io" filehandle for later use.  This later use
> of need not necessary be a "fs" operation, but could for example be
> lfs.lock(fh.fd,"r"), or asynchronous access with another Lua library
> written for this purpose.
> 
>  local path,fsdriver = fs.volpath(path)
> 
> returns "/and/file/below" in path, and the filesystem driver in fsdriver,
> so that an application knows who is responsible for dealing with
> "/and/file/below".  On the other hand fh.volume (from the fs.open(path)
> call) is the same as fsdriver.  An application can thus adjust operation
> on a specific file or filehandle of "fs".

  Again, what specific problem do you have that you are trying to solve
here?  I've come up with multiple "solutions" to problems (such as logging
locations of errors, is one) that ended up being so baroque and tedious to
use that I've abondanded them.  

  -spc (And besides, there's already a few Lua modules out there to manage
	directories ... )

[1]	I think this is the reason for the rather unusual properties of
	strncpy(), which may not include the terminating NUL byte.