Lua Library Workshop

wiki

Lua Library Design Workshop at Harvard, Tuesday, February 19

We anticipate working from about 10 to 4, with a short break for lunch. This is intended to be a working meeting, not a day of talks and panels, and we hope to emerge with very concrete, specific suggestions for the Lua team. For that reason, enrollment will be limited to about 20 people.

Please let us know if you would like to participate in this workshop. Send email to Christian Lindig [lindig@eecs.harvard.edu]. We would like to hear from you by Friday, February 8.

-- ChristianLindig?

ReubenThomas continues: Before starting it is important to establish the scope and purpose of the standard libraries. In my opinion, the standard libraries should

• provide only widely needed functionality
• provide only functionality that can't be written in Lua or would be too inefficient in Lua
• use only standard ANSI C library routines (other than Lua's own code, of course), in a completely portable manner

ChristianLinding? originally listed some recent topics from the mailing list below, most of which are probably not suitable for discussion at the workshop; here's the list, with explanation:

• Dynamic Loading of libraries to avoid building the Lua interpreter for each application. Non-ANSI
• An IO library that allows to query file and directory attributes. Non-ANSI
• Some form of coroutines or asynchronous execution. Non-ANSI; already largely addressed in 4.1
• Standard IP sockets interface. Non-ANSI
• Interface for UNIX system calls (like, exec, kill, waitpid, etc.). Non-ANSI; poslib does this but needs updating to 4.x
• Name space management to allow libraries to co-exist. This was a point about Lua libraries, not C libraries
• A buffer library. The C code already exists in auxlib; this should indeed be discussed.

RiciLake interjects:

I agree with ReubenThomas that the standard libraries should only provide services which could be written in portable ANSI C. However, it also seems clear that many of the enhancements suggested by ChristianLindig? would be useful to many people, and indeed have been requested many times. There is perhaps a middle ground here: define the standard interfaces to a set of facilities commonly available on many operating systems and possibly provide a reference implementation for one of them. A well-thought-out set of interfaces is often the hardest part of library design.

This would at least allow people to write semi-portable code; that is, code portable to platforms for which the standard interfaces have been implemented. Candidates for such interfaces would probably include IP sockets, file attributes, a subset of the rest of POSIX, and dynamic linking facilities, to select a few things from Christian's list.

Ports of the standard interfaces would be highly encouraged, but with some moral suasion against over- or under- implementing them. The spirit would be "if you have the facility, this is how you integrate it into Lua." Then a simple existence test for the implementing function would suffice to know whether the facility was available; it would be nice if it were never necessary to perform tests by operating system. This is somewhat limiting but not as limiting as pure ANSI C (and, in fact, compatible with the current philosophy where the math or io libraries may not be available for particular embedded platforms.)

The one thing I forgot from when I wrote this the first time: it would be really nice if the math library made it possible to use IEEE-754 arithmetic on platforms which support it. It wouldn't break my heart if traps were not supported, but it would be nice to be able to test and set the IEEE-754 mandated exception flags, and to have a way of at least turning traps off if they cannot otherwise be supported. And of course to test for NaN and +-Inf.

Back to ReubenThomas...

Here are some more high-level ideas of mine:

1. Is it possible to expose more of Lua's internals as libraries?

In particular, it'd be really nice to be able to get at the internals of the compilation process, e.g. to be able to manipulate code as data (rather than resorting to string mangling as at present). Similarly, see my past proposals on the list about being able to plug in a different regexp engine or (more controversially) garbage collector.

2. What about reimplementing some of the standard libraries in Lua?

Some functions, like assert, could easily be implemented in Lua, and implementing them in C gives little or no advantage. In addition, there are functions written in Lua that it is most useful to add pre-loaded to the interpreter, e.g. require (to 4.0). Having a framework to build Lua libraries into the executable would be easy (bin2c already does the work) and useful. Reducing the amount of code written in C would improve usability and reduce maintenance.

3. What else is needed?

As far as the standard libraries go, this reduces to the small question of whether there are any missing bits of the ANSI C libraries that can usefully be exposed to Lua, and the rather bigger question of whether there are any techniques that, like pattern matching, would both benefit from a C implementation and be widely used. The only (pure ANSI) functionality I've ever needed to add has been bitwise logical operations (my library for this is widely packaged, so others seem to agree).

Finally, an obvious next step if you accept the argument that as little as possible should be written in C is:

4. What about Lua standard libraries written in Lua?

I'm not talking here about a large rich set of libraries as in the StandardLibraryProposal, but just having some core functionality built into Lua. Candidates for this include functions such as assert and require which neither functionality nor performance reasons force to be implemented in C.

One specific idea: it would be useful to have a fuller interface to strftime so that one can, for instance, use it to construct lists of the names of the months or days in the current locale. Being able to pass nil (NULL) to setlocale would also be useful.

List Implementation (NickTrout)

The implementation of lists using Luas tables ( tinsert(), tremove() and getn() ) involves adding "n" to each table to hold the highest index of elements included. This addition has a few problems :-

• Name conflicts. ie. n maybe an element in the table.
• Iterators iterate over n :-

list = { "a","b","c" }
tinsert(list,"d")

for _,v in list do print(v) end -- will print a,b,c,d,4
foreach(list, function(i,v) print(v) end) -- will print a,b,c,d,4
for i = 1,getn(list) do local v=list[i] print(v) end -- will print a,b,c,d
foreachi(list, function(i,v) print(v) end) -- will print a,b,c,d

The first method for iterating over a list is clear and convenient but fails due to n. The last two methods do work, but I don't think they are as concise (or efficient?).

Now that unified method tables have been added wouldn't it be good to deprecate tinsert, tremove and getn and add a table() method which creates a table list with functionality similar to PythonLists and PythonDictionaries combined. It would probably be called "table" as opposed to "list" or "dictionary" as the functionality is a union of the two. eg.

t = {}  -- table with no methods, "clean"
t = table()  -- create table with tag/unified methods attached.
t = table{ a=1,b=7,c=8 }  -- constructor with dictionary/associative items.
t = table{ a=1 ; "b","c" }  -- mixed items.
t = table{ "a","b","c" }  -- constructor with list.

t:append("d")
print( t:len() )
t= t:sort( function (a,b) return a>b end )
print( t:count("b") )
t:print()  -- print items (rather than "<table>") and/or "print( t:tostring() )"
t:keys():print()
t:values():print()
for _,v in t do print(v) end  -- and no "n"!

Some thoughts of PeterPrade about the standardlibrary:

• I think the namespace issue should be adressed and a method shown that is encouraged to be adopted by everyone writing a library for lua. (like the simple addition of a wrapper table - for example "str" for strings library - "str.find" instead of "strfind" - although i don't know if i like the additional runtime cost. if you don't like that additional cost, naming conventions would do the job nearly as well (for example "str_find")
• I think the behaviour of the standardlibrary should be fixed to be able to write simple lua wrappers (no lua standard library function should make use of the number of parameters given to it, or should treat extra nils equally to no extra parameters given) also see TrailingNilParameters - this is important because many people will adopt the style of the standardlibrary.
• add some more tools for table manipulation (split + join (SplitJoin), clone, map(map a table using a function), collapse etc.) maybe even add a standard way how to make tables persistent (a really naive way to do it is PersistentTables) and fix the getn() issue (LuaTableSize)!
• add some tools to files manipulation (readfile+writefile of PetersStdLib - i found it very useful to add functions that take a filename and return the contents of the file in a string or that write a string into a file, this makes code just a lot more concise)

Instead of going this way, I think it's better to just accept Lua's lists as "raw lists". These raw lists are used by the VM for function arguments, and of course are the result of table constructors. The implementation is exposed (the n field). Many applications will be better served by a higher level list implementation, as you propose. But certainly you'd want the higher level list implemented on top of whatever class system you were using. I don't think the Lua distribution should be imposing a class system. So this kind of thing belongs in a bigger "on top of Lua" library... --JohnBelmonte

Some sort of basic table functionality, n, will need to stay, so that for i=1,getn(t) ... end will work for the most basic scripts. In any event, adding n to the table is not popular and should be removed. Whether the above table functionality comes as an optional component of the standard library or part of an extended library is the topic of discussion. I can see your point about it imposing slightly, but since it is very basic I cant really see it imposing too much to the extent that its the basis of a class system. --NickTrout

Summary From the Workshop

The workshop started with a public talk by Roberto Ierusalimschy on Lua. He stressed the design philosophy of Lua. The talk also made obvious that Lua's small size and fast execution is a result of a well thought out implementation.

After the talk the following people attended the Lua Library Workshop:

• Allyn Dimock <dimock at eecs.harvard.edu>
• Christian Carrillo <carrillo at tenebril.com>
• Christian Lindig <lindig at eecs.harvard.edu>
• John D. Ramsdell <ramsdell at linus.mitre.org>
• John Dias <dias at eecs.harvard.edu>
• Jon Eddy <eddy at eecs.harvard.edu>
• Noam Zeilberger <noam at eecs.harvard.edu>
• Norman Ramsey <nr at eecs.harvard.edu>
• Roberto Ierusalimschy <roberto at inf.puc-rio.br>
• Sukyoung Ryu <syryu at eecs.harvard.edu>

Here is an incomplete list of topics that were discussed at the Lua Library Workshop. Maybe the other attendees of the workshop can help to complete it.

• I took a look at the IO libray of Lua 4.0 and proposed some changes. The essence of the proposal is to remove functions like writeto which handle files implicitly. The rational is that they are not general enough when you deal with multiple files. The discussion was mostly around less radical ways such that the casual user still can have default input and output channels.

http://www.eecs.harvard.edu/~lindig/papers/lua-iolib/

http://www.eecs.harvard.edu/~lindig/papers/lua-iolib/slides.pdf

• A big topic of discussion was name space management. A hierarchical name space like IO.open allows everybody to contribute libraries and simplifies the transition between libraries. For example, at this point is it not clear whether libraries should be written in an object-oriented style, or a procedural style. We could have both and pick as a standard what emerges later. Problems related to name spaces are:
  • Writing module X.Y.Z, how do I refer to function X.Y.Z.f, without having to know that this is X.Y.Z? A local module M does the trick.
  • How are modules mapped into the file system? The success of Python suggests to use directories hierarchies. Discussion was around how to load libraries and search paths. What can be written in Lua, what must go into the kernel?
  • Inside a module, is there a way to "open" other modules such that their members can be accessed without dot notation? Seems to be difficult.
• Loading of object files at run-time. Consensus was, that this is not a technical problem, but a problem of policy. Where to put the tiny target specific module that implements it? John Ramsdell suggested to mine Python and the GNU Libtool for implementation details.
• A mechanism that forces variable declarations in large projects. A flexible technical solution emerged that I did not fully understand. Probably Roberto can explain it.
• Christian Carrillo suggested some changes to the C API to support Lua threaded environments better. The details escaped me because I don't write threaded programs.

-- ChristianLindig?