Standard libraries (was Re: Virgin tables)

[Mark Hamburg <mark@...>]

On Dec 30, 2010, at 7:40 AM, Lorenzo Donati wrote:

> I'm not saying that a more complex/feature-rich/safe data structure is never needed, but, as I said some time ago in another thread (but I was not the only one), maybe the problem with all this fuss over holes in tables and the like is that most people seem to see Lua tables as an high level data structure (some time ago I thought it too). Indeed they are, if compared with basic data structure of most common languages (C/C++ arrays, Java arrays, for example).
> But in a Lua perspective, IMHO, tables are best viewed as a low-level facility (even if very powerful), on which to build more complex/safe/user-friendly data structures.
> 
> The fact that a newbie can do amazing things with tables doesn't mean that they should be "idiotproof", as some Java data structures are (well at least a more clever idiot enters the game :-) )
> 
> Since it is the only data structure of Lua, the balance between ease of use, efficiency and readability attained is a benefit I wouldn't forfait. And as I learned reading this list, Lua authors have spent years to achieve that.
> 
> Instead of pushing for changing such a cleverly crafted aspect of the language, I would really spend time in advocating a standard collection of general-purpose library. I think that newbies (if they are the target of this thread) would better be served by an "almost endorsed" library with all the basic tools one usually find in other languages (say pure arrays, maps, sets, string manipulation functions, etc.).
> 
> This would smooth considerably their learning curve. I think that for a newbia is more frustrating to have the usual (on this mailing list) reply "but you can do that with these two lines of Lua code..." followed by some idiom that leaves one puzzled for a while (see tricks with 'gsub ' or 'match') (please don't beat me with a stick, I actually got to love some of these tricks _now_ ;-) :-) )

The more I think about it, the more I think the appropriate agenda here (if there is one) is not to change the behavior of Lua's core constructs (at least not initially), but to:

1. Define a set of standard higher-level constructs. I concur with Lorenzo that tables feel like high-level structures but they really are the CONS-cells of Lua -- i.e., the basic building blocks and highly-useful in themselves but "basic". So, start wrapping them up into high-level structures like arrays (which can be hole friendly), sets, etc..

2. Start encouraging use of these constructs.

3. While trying not to warp the language, start tuning the implementation and/or the language to make using such constructs clearer more efficient both from a textual standpoint and from a runtime standpoint. For example, "mySet[ x ]" is going to be a faster test for membership than "mySet:hasElement( x )" but it relies on the internal implementation of the set construct and won't work for sets that need to include nil or NaN. Can the implementation of Lua elide the runtime costs associated with the method dispatch (I'm pretty sure LuaJIT can)? Are there things that should be done syntactically to make this easier? For example, an __index metamethod could allow the syntax mySet[ x ] and could even work for nil and NaN, but then we get the wrong result when one wants to add the string "iterateElements" to the set assuming that sets have a method named "iterateElements". To make that work, we need to actually treat the colon operator as a real feature of the implementation (OP_SELF essentially makes it one anyway) rather than simply syntactic sugar.

Unfortunately, the process of defining standard libraries runs into the ground whenever it comes up. Since this seems to be beyond the scope of what the core Lua team wants to tackle, what would be good is if they could at least settle on some designated individual or two on whom they could apply some form of official blessing. This choice might be based in part on submitted proposals or code as opposed to simply blessing someone. But the point would be that the resulting libraries had been discussed within the Lua inner sanctum in addition to being discussed on more public forums like this list and through that they gained more specific approval and support.

Mark