Re: new "empty" value/type in Lua?

[Tim Hill <drtimhill@...>]

Good points from all, but I don't think I can agree.

I think the point here is that there are different degrees/scopes to standardization:
1. At the bottom, there is the "per project" conventions. The work-arounds I mentioned, as well as Andrews's, fall into this bucket. Yes, a single empty table could act as a flag WITHIN a single Lua project or program, but...

2. Next, you have informal conventions that are often embodied in "best practices" or accepted ways to do things. Much of the Lua module system works like this.

3. Next you have the standard runtime/library.

4. Finally, you have the enforced conventions of the language itself.

To my mind you try to stay as far "up" that stack as you can. So why am I arguing for something all the way down at #4? Because at every other level any solution is scoped wrongly. Yes, for a given project I can use an empty table as a special placeholder, or (as per the other suggestion) I can use metatables to "hack" sparse arrays (though this is harder than it might at first appear, at least to do efficiently).

But what if (as I am currently doing), I am coding a general purpose C library that has to handle polymorphic arrays? How do I project "empty" behavior "up" into Lua? I *could* do it with a variety of more or less elegant hacks. Then someone else comes along with the same problem, and they solve it in a slightly different way. Then a third … and so on. Pretty soon you have a big mess on your hands. This is the classic balancing act: too many standards and everyone is handcuffed, too few and you create chaos.

You see, when you move down that stack above, you are going from a single local project to a broader and broader scope, until at #4 you reach everyone. At the top, hacks are acceptable (as was stated in a reply). But the further down you go, the cleaner you need to be. Some of the stuff I'm working on is hovering between #2 and #3, and I for one am uneasy of level #1 hacks drifting down that stack too far. Hence my desire for a much less hack-y way of doing things.

To take a canonical example, what is a clean way to surface a database result set that might include NULLs? In a standard library that many people are likely to use? Yes, there are a dozen ways, but I don't regard any of them as clean, and when I analyzed the problem, it really came down to this lack of a storable empty value and the overloading of nil.

And, to be honest, I *do* think Lua lacks the concept of empty; the current Lua documentation has to kind of wriggle a bit when explaining arrays precisely because you cannot have an empty array element. Even after 3 years of working with Lua I still find this one of the "oh really?" bits.

--Tim




On Jun 27, 2013, at 8:40 PM, Andrew Starks <andrew.starks@trms.com> wrote:

> On Thu, Jun 27, 2013 at 8:28 PM, Rena <hyperhacker@gmail.com> wrote:
>> On 2013-06-27 8:57 PM, "Tim Hill" <drtimhill@gmail.com> wrote:
>>> 
>>> I'm going to get shot for suggesting this… (hides under bed)
>>> 
>>> I've been writing a lot of Lua code recently, some of it quite generic in
>>> nature. One thing I frequently come up against is the need to handle data
>>> polymorphically in tables. And an effect of this is to hit against the
>>> overloaded use of "nil" in Lua.
>>> 
>>> It seems to me that "nil" is overloaded; it means both a noun: "no value"
>>> and a verb: "delete this table entry". Now, most of the time, the delete is
>>> hidden since reading a non-existent table entry returns nil. So having an
>>> explicitly stored nil versus having no value stored at all is equivalent
>>> (all this I love, btw … 100 points to Lua).
>>> 
>>> But it all rather breaks down with arrays (sequences). I cannot have
>>> sparse arrays, which in Lua terms means I cannot have "nil" in an array
>>> element. So, when writing polymorphic code, how do I handle cases (which,
>>> trust me, i DO have to handle, with "no value" in an array element? There
>>> are, of course, a whole bunch of work-arounds: Use a sentinel value; use an
>>> empty table (assuming empty tables aren't value data, and ignoring that each
>>> one is unique); use a magic userdata (assuming I have access to C code to
>>> make one); etc etc.
>>> 
>>> But, let's face it; they ARE all workarounds. I think there IS a case to
>>> be made for being able to store a value in a Lua array that is uniquely NOT
>>> any other Lua value. Essentially, I think that value is "empty"; a valid Lua
>>> value that is simply not the same as any other Lua value (including nil).
>>> 
>>> Basically, empty would act like nil in most regards, except:
>>> -- Storing empty in a table element actually stores the value, rather than
>>> deletes it.
>>> -- When used in a boolean context, the empty value evaluates to true
>>> (unlike nil).
>>> -- Has a type() of "empty"
>>> 
>>> Other arguments in favor of this:
>>> -- Judging by the mail list chatter, a lot of Lua beginners struggle with
>>> the problem of nil in sequences.
>>> 
>>> -- Better impedance matching to SQL NULL values; since empty can bridge to
>>> a SQL NULL when a Lua array is used to hold a SQL row.
>>> 
>>> Thoughts anyone?
>>> --Tim
>>> 
>>> 
> 
> I think that you can have sparse arrays.
> 
> As I'm sure you're aware, you can modify __ipairs  and __len in any
> way that you wish. So a metatable that marks the holes or provides the
> length can be used in your version and then BAM! Sparse arrays. Heck,
> you don't even need any storage. Just use make an __ipairs function
> where the factory uses pairs to find the max int, picks the biggest
> index and then poops out an iterator function that keeps going until
> math.max is reached.
> 
> 
> The implementation doesn't matter. I've posted emails just like yours
> and like you, lately I've been doing more writing in Lua than writing
> about it. The more that I use it, the more that I've come to
> appreciate that it just isn't made to be the most convenient language
> in the world. You end up doing quite a bit of typing, or you can use
> its mechanisms to get what you want, ala how Penlight does it, which
> probably has a sparse array module....
> 
> So, methods, not policies. Those work arounds that you've mentioned
> are _how_ you do it, in Lua. That is the way. No need for yet another
> "thing."
> 
> As an example, in my little library that I've built up for my project:
> type looks at __type, error and assert use LuaLogger as well as
> string.format, dogs hang out with cats, math has is_int, table is
> overloaded with Penlight's tablex, coroutines can be made with tables
> that have a __coroutine metatable field, and all manner of other
> fluffery abounds. It's all good because I put it there and that's how
> you're supposed to do it, if that's how you choose to roll... at least
> that's what I've come to understand.
> 
> -Andrew
>