Hello,
I've been writing Lua and C libraries for this wonderful little language for some time. However, a recurring criticism I have is regarding the C ABI compatibility. The official Lua site details this quote: "Different versions are really different. The API is likely to be a little different (but with compatibility switches), and there is no ABI compatibility: applications that embed Lua and C libraries for Lua must be recompiled.".
I can't help but suggest Lua changes this behavior in the future. Lua is a boilerplate language in the sense that every user is provided the basics to operate, but in thousands of codebases Lua is used for user-to-user unique purposes, like networking, data handling, et cetera. There's millions of downloads the past few years for C modules that implement this functionality into the language.
As a necessary evil from a small standard library, this makes C modules very popular. Lua by no means can predict what each user may need, so the user should just install a module.. that makes sense, but the ABI compatibility means not every user can do this easily, or in some cases the user simply cannot do this. Even regardless of what Lua may be intrinsically designed for, I believe it's necessary for the language to continue adapting to how so many users are using it, within reason.
If Lua is going to rely on C to extend the unique desires of each author, then Lua needs to boast extensive ABI compatibility. If Lua has what the user wants, but Lua cannot provide third-party functionality necessary, then despite all the pros, another language is going to be used, or the user will be forced to manually write and maintain their own module (which is often a pain for quick tasks, a large turn off).
I've personally criticized some major breaking changes in the ABI, like the removal of luaL_register. I did not directly ask about this, but I found an old mailing list post detailing that this change was made to discourage people from exporting their module into the global environment. If this is true, I find this one example of a significantly breaking change (making most C libraries incompatible with Lua 5.1+) to simply reflect a (albeit healthy) coding practice. Instead of this, documentation could've been improved to passively coerce C module authors and it'd create a similar benefit without breaking compatibility. I cannot use the LuaSec, LuaSocket, or lua-cjson modules largely because of this change. And these are massively popular modules.
In my own experience, there's been many times I need to read JSON, make a network request, call FFI functions, and furthermore. All of these operations have libraries upon libraries in C to provide this for me, but none work because an ABI function was renamed, slightly modified, or in justifiable cases overhauled. So I quite literally abandoned Lua for these projects because I couldn't be tasked at the time with hacking these modules together or writing my own.
Lua 5.3's luaconf.h file provides useful compatibility macros for users to tailor but I deem this not practical because end users may not be technical enough, may not hold the motivation, or may not have the administrative ability to recompile Lua for the specific task. But this is certainly better than nothing.
For example, the LUA_COMPAT_BITLIB compatibility flag in Lua 5.3 allowed people to write 5.2-5.3 cross-version bitwise operations. But once bit32 was removed for native bitwise operator support, this became impossible in Lua 5.4, because the Lua 5.2 & 5.1 compiler would simply reject any foreign operator. (Yes, you can manually call metamethods, but this is slower and messy). It's true the language must move on, but the embedded purpose of Lua means once a codebase implements a version, it's likely never going to update.. because that'd break thousands of community-made scripts. But we still want our C modules to be available for this massive base of users. Perhaps not for 5.1 now, but to not make the same compatibility mistakes in the future. If Lua 5.6 or Lua 5.7 may ever exist.
Some conference discussions of Lua help me infer there is an increased interest in compatibility across the board. I was advised to write an extensive email by my peers, because they are certain it'd sway the development of this language. I only hope that ABI changes from here on out will be extensively held with regard and skepticism, and furthermore that we may see 5.3 compatibility macros implemented into 5.4, and potentially 5.5 (i.e, LUA_COMPAT_BITLIB is very useful, since you cannot write a cross-version bitwise operation in Lua 5.4 currently, LUA_COMPAT_MODULE could solve C lib import incompatibility for technical users).
I may very well be biased in my observation and criticism of implementation detail but I cannot help to share my thoughts. I am interested.
