Re: Empty? No. Array? No. Has? Yes.

[Andrew Starks <andrew.starks@...>]

On Sat, Jul 6, 2013 at 2:33 PM, Jay Carlson <nop@nop.com> wrote:
> On Jul 5, 2013, at 2:54 PM, Andrew Starks wrote:
>
>> On Thursday, July 4, 2013, Jay Carlson wrote:

snip

>> Would some kind of mechanism to discover the existence of a variable suffice? That is, the mechanism returns a boolean, not a void / etc.
>
> I apologize for nitpicking:
>
> I don't think you're asking the question you really mean. Whether a variable exists is not particularly interesting.
>
> For local variables, the compiler can determine the existence of a variable. In fact your editor could do this, and some do. Local variables are names for slots, and the slot always exists. The slot may contain true, false, nil, or any other value. But it exists.
>
> Global variable references are syntactic sugar for gets and sets on a per-chunk table (called _ENV in 5.2). In many ways it's just another local variable. Asking whether _ENV itself exists is not interesting.
>

I appreciate the explanation and your graciousness. I'm on a
particular train of thought that is getting no traction. I'm convinced
that the reason is that there is something obvious that I'm missing or
obviously unimportant that I think otherwise. That thing is SO
obvious, that people may be too polite to point it out and any time
that someone is willing to attempt it, I'm richer for it.

I need to think on what you've written a bit more. I don't think that
your points of fact are misunderstood by me, except that tricky "not
interesting" part.

So, yes names are slots and anything can be in them. But it simply
isn't true that "not ever there" and "nil" are the same thing. As Joel
pointed out, "
All non-trivial abstractions, to some degree, are leaky." Here's some
stupid code:
````
print(table, math, string)
--> table: 0x7fc2d8c059b0   table: 0x7fc2d8c074a0   table: 0x7fc2d8c06b10
local table = nil
local math = math
_ENV.math = nil
string = nil

print(table, math, string)
--> nil table: 0x7fc2d8c074a0   nil

do
    print(math) --upvalue
    --> table: 0x7fc2d8c074a0
    local math = _ENV.math --now nil and gone forever
    print(math)
    --> nil
    print(table) --upvalue
    --> nil
    local table = _ENV.table
    print(table) --now back
    --> table: 0x7fc2d8c059b0
    print(string, _ENV.string)
    --> nil, nil
end
print(table, math, string)
--> nil table: 0x7fc2d8c074a0   nil
````
So, clearly setting things to nil, in the context of upvalues is
significant and there is a very big difference between "nil" and "not
there," all the up the call stack until the Great Table In The Sky
(_ENV) is checked.

and then there've my current favorite example:

````
local f = function(...)
    print(select("#", ...))
end
f(nil,nil,nil)
-->3
f()
-->0
f(nil, nil, (function() end )())
--> 2
````
and of course `type()`, which is an error.

So here we can see a bunch of cases where `nil` is absolutely "a
thing" that is separate from "not ever having been defined." And I
think that's great, and at the proper level of abstraction, I can see
how it might be irrelevant.

But they still are not equivalent, not matter how hard you squint your
eyes. They're not even equivalent in daily life, even for normal
people.

And then we have this implementation reality:

````

local array = {}
collectgarbage()
local k, b = collectgarbage("count")
local mem_start = math.floor(k) * 1024 + b

--Memory before loop / array elements:
print(mem_start)
--> 24858

for x = 1, 2^16 do
    array[x] = tostring"x"
end

for x = 1, 2^16 do
    array[x] = nil
end

collectgarbage()

local k, b = collectgarbage("count")
local mem_end = math.floor(k) * 1024 + b
print(mem_end,  (mem_end - mem_start) / 2^ 16 )
--> 1073434     16

array  = nil

collectgarbage()
local k, b = collectgarbage("count")
local mem_t_release =math.floor(k) * 1024 + b

print(mem_t_release,  (mem_end - mem_t_release) / (2^ 16  - 3))
--> 24906   16

````
So clearly lua stores nils, even inside of tables. And not just a
little. I mean, it really holds on to them and won't collect them,
even when you ask it to; or at least until the container is collected.

And so, they are there and it remains interesting, to me, in that this
would be a path in which, if you could only ask a table if the index
was undefined or not, you could know something more than you otherwise
would. Something like `rawget(t, i)` returning 0 values when nil was
not present would be enough to get you 70% there.

Then you could either hang yourself with that knowledge, or at least
create some kind of hard to find (but presumably uninteresting) bug in
some array library that you were trying to write.

But again, I'm most likely thinking in a "passion beyond reason" mode
and in the end, I still like Lua the way that it is.

-- Andrew



> It is very nice that Lua only has one aggregate type. But it does mean that changes to many different kinds of language behavior reduce to the behavior of the table type.
>
>> I thought Lua did keep nils in tables around internally, and just didn't provide a way to determine the difference between "not there" and "not there, as far as you know," As it does with local variables.
>
> I agree with Tim Hill: nil assignments do not distinguish programmer intent in a couple situations they expect it to.
>
> Let's take the following code, applied to an array-like table:
>
>   i = #t
>   t[i] = nil
>
> This could mean three things:
>
>   1) Shorten t by 1 by deleting the last element of the sequence. It becomes shorter.[1]
>
>   2) Make future references to t[i] return nil; the length is not important. (It's just used like a table.)
>
>   3) Make the sequence member t[i] return nil; that is, store nil but take no action on length.
>
> Note that metatables can help for a fixed length array; __len can always return (say) 256, always producing behavior 3. But there can't be a shrinkable array.
>
> So, for total overkill, I introduce new syntax to make intent clear:
>
>   i = #t
>   delete t[i]
>
> This definitively means case 1: shorten the sequence. I am not sure what the correct semantics are for any element other than t[#t]. But in the t[#t] case, after that code block #t == i-1. This means there must be a hidden length variable somewhere, akin to the old .n. Let's call it t.# for now. (We can make it an lvalue too.)
>
> The hidden t.# must be created when "t[i]=nil" is evaluated, if i==#t, and t[i] previously contained a non-nil value. This covers the situation where #t would decrease in Lua 5.2 but should not if our intent is to store a nil element, not delete it.
>
> I am not certain what should happen to t.# in other cases. One obvious thing for "delete t[2]" to do is shift t[3], t[4],... down. But that requires core Lua to magically detect array-like tables with positive integer indices and perform an O(n) operation from the table library. That doesn't sound good.
>
> There are non-array uses for "delete" assuming it triggers a separate metatable function. In particular, it allows environments and struct/class implementations to blow up on undeclared elements, yet still allow a distinction between storing s.callback=nil and undeclaring s.callback.
>
> Finally, note that this is quite conservative in several ways. It does not create any new value types, nor does it affect existing non-array code. The hashtable implementation does not change. There is still no way of writing "if undefined(t, 'foo')"; any such protocol is by private agreement with a class. The only thing significantly changed is the notion of sequence length, which only indirectly allows an undefined() predicate in the case of sequences.
>
> Jay
>
> [1]: Note that in the present implementation a sequence represented by t may get *much* shorter after the t[#t]=nil case:
>
>> t={1,2,3,nil,nil,6,7,8}
>> return #t
> 8
>> t[#t]=nil
>> return #t
> 3
>
>