Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

[William Ahern <william@...>]

On Thu, Jan 12, 2017 at 06:57:52PM +0000, Dibyendu Majumdar wrote:
> Hi William,
> 
> On 12 January 2017 at 01:18, William Ahern <william@25thandclement.com> wrote:
> > OTOH, Lua wouldn't be the language it is today if backwards compatibility
> > was a higher priority. Features like closures over mutable variables and
> > stackful coroutines wouldn't have been developed; features which arguably
> > are now defining characteristics of Lua relative to other languages. Other
> > than Scheme, I can't think of any other non-academic language with similarly
> > powerful constructs. And that's largely because before they get there
> > language designers and implementers decide their language is "good enough"
> > and switch into long-term maintenance mode where, at best, only pale
> > imitations of the above can be added.
> >
> 
> I am relatively new to Lua so my knowledge of Lua history may be
> flaky, but I thought these features you mention were introduced in Lua
> 5.0 in 2003? So they can hardly be justification for incompatibilities
> between 5.1, 5.2 and 5.3. Or am I missing something?
> 

Right, but it was a huge break from 4.0.

The big break in the 5.x family is _ENV, and the related removal of module
and function environments. Most changes in 5.2 and 5.3 are feature additions
that had no analog in previous versions. Most removals were in the C API and
easily emulated.

While the move away from module was received well, I think most people felt
that replacing function environments with _ENV was more of an annoyance and
needless incompatibility. But both function environments and module() were
effectively experiments with the semantics of free variables, not merely
pragmatic solutions to utilitarian global scoping and sandboxing issues.

I don't know if the Lua authors realized so clearly beforehand, but I think
their invention of lexical _ENV shows that's how they were conceptualizing
the issue some time after 5.1 was released. _ENV was such an annoyance
exactly because it forces you to change your thinking (not just your code)
about what you're trying to accomplish when fiddling with global scoping and
sandboxing, and how you should accomplish it. It shifts the emphasis away
from implementation details and back to the structure of your code.

_ENV is what you get when you realize more clearly (undistracted by
implementation details and immediate problems) that both global scoping and
sandboxing are simple namespacing issues, and in Lua all namespacing issues
could be solved like a lexical scoping issue--with blocks and local. This is
so because of Lua's dynamic typing and the unifying role of tables in the
language, along with their metamethod capabilities. Lexical _ENV is a simple
construct that replaces an out-of-place interface to the VM; namely,
setfenv.

What I'm trying to say is that I would disagree that there weren't any
changes between 5.1 and 5.2 as significant as closures and coroutines. I'd
argue that _ENV is a much more important construction from a language design
perspective than usually thought, and whether or not time and experience
proves its value nonetheless worthy of breaking backwards compatibility.

>From my perspective _ENV seemed to come out of the blue. Who knows what the
Lua authors might pull out their hat, or what might fall into their lap.
That's why I don't think I'm qualified to be able to say that Lua 5.3 is
"good enough". Maybe the Lua authors might devise a thread-safe solution for
moving or referencing objects between Lua VM states which requires an
incompatible change to the behavior of tables, similar to how Lua 5.2
disallowed lazy assignment of __gc metamethods. There are lots of hard
problems like that worthy of incompatible changes.