Re: Strong tables in Lua 5.4

[Hisham <h@...>]

On 17 March 2018 at 03:32, Philipp Janda <siffiejoe@gmx.net> wrote:
> I think that storing nils in tables is worth a break in backwards
> compatibility. We should just call it Lua 6.0 and design a nicer syntax for
> deleting keys.

I fully agree with this.

Lua 6.0 with nils-in-tables will be an excellent language for new
green-field projects (scripting for the next big game, etc.) but there
is no viable "path for compatibility" that makes a language with
nils-in-tables fit in the Lua 5.x ecosystem.

I think this is the main disconnect (among many) in this whole
conversation: considering ecosystems.

Every `t[k] = f()` of every module and dependency (and their
dependencies recursively) would have to be checked for memory leaks.
And this won't be the typical Lua 5.x to Lua 5.(x+1) situation where
you try out a dependency on a new Lua version and it doesn't build
because lua_dump now has an extra argument, or lua_objlen was renamed
to lua_rawlen in a Lua/C module, or you get a crash with a stack trace
in your testsuite because setfenv() doesn't exist anymore. Patches to
add Lua 5.(x+1) support in a module are typically small and easy to
write.

Every attempt of smoothing out the incompatibility ("strong tables",
undef syntax hack, etc.), while well-intentioned, just adds terrible,
terrible pitfalls to the language (e.g. "Of course you may not pass a
strong table to old libraries").

Adding nils-in-tables is a bold move and everyone seems to agree it
moves the language forward. But it is a major breaking change, on par
with the removal of old-style %v upvalues and introducing proper
lexical scoping. If this is added:

* it won't make sense to call this anything other than Lua 6.0
* any attempts at making a compatibility bridge will mean that the
entire existing Lua 5.x ecosystem is "silently broken by default" on
Lua 6.0
   * the entire existing Lua 5.x ecosystem will not embrace the bridge
   * the parts of the ecosystem that do will not do so at the same time
   * this will bring major grief to users

In light of this, breaking the _existing_ syntax would be _helpful_ to
ensure things are not silently broken by default.

Let's be honest: Lua's 5.x ecosystem is small and fragmented, compared
to other languages. There's the LuaJIT situation, which, even if you
choose to ignore it and consider it a different language, affects the
way developers mantain code for Lua. Developers invested in a
particular Lua 5.x version or LuaJIT will support Lua 5.y > 5.x _if
it's not too much work_ (I've heard that from people many times). They
will not audit their entire codebase and dependencies for every usage
of nil just for Lua 6's sake. Lua 5.x will stick around for a long
time, and it is what it is, kind of like ANSI C (89). However,  Lua is
in a unique position as an embedded language, and has many ecosystems,
not one. That is, many projects adopting Lua completely disregard the
existing ecosystem(s) around it. Many new projects will have no
problem adopting Lua 6. This is not a Perl 5 vs 6 situation, or a
Python 2 vs 3 one.

Will a Lua 6 (with no attempted "path-for-compatibility") weaken Lua's
already fragile Lua 5.x ecosystem, which I worked so hard on to
promote via LuaRocks for so many years? Yes. But again, it is what it
is. That ship has sailed: the Lua 5.x ecosystem is already de facto
tied to the success of Lua 5.1 and LuaJIT -- the number of Lua
5.(x>1)-only modules is tiny (only 4.8% of LuaRocks modules marked as
5.3-compatible are not marked 5.1-compatible). Lua 5.x keeps getting
better and better, but most of us never get the chance to use a lot of
the good stuff in production because we're tied to Lua 5.1.

We have a chance to do better with the Lua 6 ecosystem, without
injecting the Lua 5.x ecosystem into it in a "silently broken by
default" state. That is, if we care about an ecosystem at all. And if
we don't, breaking compatibility and making a better language without
the clever hacks is a no-brainer anyway.

-- Hisham