Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

[William Ahern <william@...>]

On Thu, Jan 12, 2017 at 12:02:49AM +0000, Dibyendu Majumdar wrote:
> On 11 January 2017 at 21:06, Hisham <h@hisham.hm> wrote:
> > On 11 January 2017 at 17:08, Roberto Ierusalimschy
> > <roberto@inf.puc-rio.br> wrote:
> >>> Please, do not make such a severely breaking change in a 5.3.x release.
> >>
> >> We did not realize that. It will be corrected.
> >
> > Thank you! :)
> >
> 
> Hi,
> 
> I was wondering if it is possible to consider making strong backwards
> compatibility guarantees for Lua from 5.3 upwards. I think this will
> mean some constraints to Lua's development obviously, but it would
> strengthen Lua 's ecosystem.
> 
> The strongest argument I could make is that Lua is built on C, and
> imagine how difficult it would have been if C had kept changing in
> incompatible ways every few years, and all new C compilers stopped
> supporting older versions of C.

OTOH, Lua wouldn't be the language it is today if backwards compatibility
was a higher priority. Features like closures over mutable variables and
stackful coroutines wouldn't have been developed; features which arguably
are now defining characteristics of Lua relative to other languages. Other
than Scheme, I can't think of any other non-academic language with similarly
powerful constructs. And that's largely because before they get there
language designers and implementers decide their language is "good enough"
and switch into long-term maintenance mode where, at best, only pale
imitations of the above can be added.

If Lua were more like Python, where backwards compatibility was sacrificed
for much less expressive, only marginally better constructs, the argument
for more stable changes would be stronger. But are you really prepared to
say that Lua 5.3 is "good enough"? Why wasn't Lua 5.1 good enough? It's an
important question because LuaJIT is fully committed to the semantics of Lua
5.1, and the LuaJIT community is probably larger than that of Lua 5.3.

If Lua 5.3 really is good enough, given Lua's track record I would expect
future versions to naturally maintain strong backwards compatibility,
regardless of any specific intent to do so. Indeed, for most problems it's
trivial to write Lua 5.1, 5.2, and 5.3 compatible code.

But if it turns out that Lua 5.3 isn't good enough--that there are more
powerful constructs or semantics that would work well in the Lua-language
family--it would be a shame to forgo that experiment.

Preserving the ability and motivation for substantial experimentation means
we sometimes get incompatibilities, like with __ipairs[1], that have
arguably poor cost+benefit tradeoffs. But there's no way to avoid those
while also preserving the ability to adopt really greats changes. The true
cost+benefit can only be known in hindsight, and so failure has to remain an
option.


[1] My beef is that __len forecloses lazy evaluation. I'd personally have
liked to see __ipairs preserved with a fallback to __len. But I can totally
understand how that might seem too awkward. And I can appreciate the desire
to circumscibe the complexity of metamethods. __pairs and especially
__ipairs were already concessions in the sense that, unlike __call or __gc,
they're trivial to implement in user code and don't directly supplement the
language semantics. __len makes alot more sense as a language addition, and
I'd have been blissfully ignorant had __len come first. I guess the
evolution of __pairs begetting __ipairs begetting __len is a great example
of how even at a micro-scale Lua iteratively becomes a much better language
by being less strict about backwards compatibility.