Distributing Lua modules (was Re: To all Lua rock maintainers (also included considerations on Lua's ecosystem and a Lua distribution))

[Sean Conner <sean@...>]

  I was scrolling through the ulua_autobuild.html page when I was struck by
the reason lua_espeak was not included: "not valid rock version '1.36r1-1'". 
I downloaded the rockspec from LuaRocks and ran "luarocks lint" over it, and
it was fine.  So LuaRocks has no issue with the version number.  I then went
back to this message, your original announcement, to add some further
comments.

It was thus said that the Great Stefano once stated:
> Hi all,
> 
> As part of my work on ULua [1], I routinely go through the task of trying
> to install the Lua rocks from luarocks.org (root manifest) on Windows, OSX
> and Linux.
> 
> Problems include, but are not limited to:
> + rocks contending the same module name, who doesn't love 'json'?
> + rocks polluting the module namespace with generic names - anyone using
> 'test'? - worsening the problem above
> + missing supported OS and required external libraries specifications

> + no automated way of building such required external libraries

  This is a huge issue and there is no easy answer to this.  Sure, you could
try the whole "download, extract, configure, make, make install" routine,
but that generally only works on Unix, not Windows.  You could try to
install the prebuilt packet for the distribution, but for Linux alone you
have rpm, yum, apt-get (and a few others I'm forgetting, like the Gentoo
package manager); the BSDs (and I think OSX) use ports; OSX has another one
(I keep forgetting it since I rarely use it myself) and I have no idea what
Windows developers use.

  Worse, you can't depend upon the platform (or distribution) to even have
the package.  Some distributions (like RedHat) are very conservative and
often times, you have to use third party repositories for certain libraries
(and again, syntax and configuration of these depend upon the package
manager used).

  There's a real reason why this isn't automated.  And before you say "just
auto update" no, that isn't an answer for one reason---breakage [10].  In
fact, I just found a nasty bug in the latest version of iOS that requires a
powercycle to fix [12].

> + no standardised build tool: make, cmake, nmake, automake...just lake is
> missing!

  Because everybody feels "make" is broken [13] and they can do better.  I
don't think so, but that's my opinion.

> I wonder how that is preferable to assuming semver and accepting that some
> modules do not follow it.

  The major contension to semver [14] I've come across is that one person's
PATCH (which is a non-breaking backwards-compatible bug fix) is another
person's breaking, non-backwards-compatible bug fix (say, they had a
workaround of the bug that now crashes in the presence of the bug fix).  The
only safe thing you can say about version numbers is that each one describes
a particular program at a particular point in time.  Windows 3, Windows 95,
Windows NT, Windows 98, Windows 4, Windows 2000, Windows XP, Windows 7 (I
think I have those in chronological order).

> More generally, the version format specification is documented [9] as
> "please look at the source code".
> Also, rocks which contend the same module but are unrelated (json4lua,
> luajson) are not rejected. All these details point to a 'lack of policy'
> which I find detrimental to the Lua ecosystem.

  There's a reason for "mechanism" not "policy".  The previous "policy" of
modules (Lua 5.1) wasn't generally liked (from my understanding) so the Lua
team moved more towards a "mechanism" that no longer polluted the namespace
while still keeping the "local global environment".  

  And for a more contentious example of "enforcing policy" one need look no
further than the Linux systemd controversy [15].  With a policy in place,
there is only *one way* to do something.  And that can rub some people the
wrong way.

> + agreement on module responsibilities (no globals ever, ...)
> + agreement on version format, on its meaning, and on the module metadata
> + agreement on an extended standard library (*everything* IO included, ...)
> + agreement on Lua compile flags (what type is a number? which compat
> options? which conversion options? ...)

  I write my modules with the asusmption of the default number type (double)
and no campatibility functions exist.  For me, that's the safest assumptions
I can make---the integer only Luas are highly specialized and my code
probably won't be used there (but I never know).  

> + agreement on documentation and tests

  I personally wouldn't mandate tests as I rarely have tests myself (bad
habit, but I won't enforce behavior on others that I myself do not do).  But
in some cases, like my "org.conman.env" module, I have no way of doing any
meaningful test.  Environment variables vary from platform to platform, and
the mechanism for setting an environment variable not only varies from
platform to platform, but under Unix, from command line to command line!

  Then there's the syslog() wrapper I have.  If it compiles, it will
probably work, but there's no way of automatically testing that.  Or rather,
there is, but the amount of code required to check would probably exceed the
number of lines of code in Lua itself!

> All this necessarily has to come from upstream. Yes, the Lua team blessing
> hand.

  Why?  They have in their mind the intended use of Lua (embedded).  This
use of Lua as a general purpose langauge with a standalone executable, goes
a bit beyond their intent.  If anything, what you need is the blessing of
Hisham, maintainer of LuaRocks.

> If the idea was "let's leave this to the community: a natural selection
> process will yield an optimal result", then it clearly has not worked.
> 
> I am happy that Lua is light and customisable and that this contributes to
> its success in embedded systems and as a glue Language.
> It would be great if steps were taken to offer also a pleasing experience
> as a general-purpose programming language.
> This would take the form of a batteries-included version (side by side to
> the light version already available) plus answers to the +s above.

  Short of co-opting Hisham into this, then just state the criteria for the
uLua module distribution, which seems to be:

	* Semantic versioning
	* No external libraries
	* latest Lua Rock available for Linux, OSX and Windows
	* Unique module name without dots, alphabetical order per LuaRock
	  name

  I have no problem with opinionated software, as long as I know the
opinions up front.

  -spc 

> [1] http://www.scilua.org/ulua.html (a binary Lua distribution for Windows,
> OSX, Linux)
> [2] http://www.scilua.org/luarocksorg/state
> [3] http://www.scilua.org/luarocksorg/state/pass.lua
> [4] http://www.scilua.org/luarocksorg/state/error.lua
> [5]
> http://www.scilua.org/luarocksorg/state/target_os/target_arch/error_sys.lua
> [6]
> http://www.scilua.org/luarocksorg/state/target_os/target_arch/excluded_sys.lua
> [7] http://www.scilua.org/luarocksorg/state/target_os/target_arch/log
> [8] http://article.gmane.org/gmane.comp.lang.lua.general/112719
> [9] https://github.com/keplerproject/luarocks/wiki/Rockspec-format




[10]	Earlier this year I tried upgrading to OSX Yosemite (or whatever the
	first non-cat version [11] of OSX was) and inside twenty minutes I
	was back to the previous version---it broke my workflow *that much*.

[11]	What a *stupid* way to version software.  Grrrrrr ...

[12]	Thank you, auto updating iPhone!  At least I know what triggers the
	bug and can avoid it.  But it's a bug that broke an existing working
	gesture.

[13]	The Windows make is "broken" in that it's too simple.  GMake is much
	better and is more capable than people realize.  Here is what I need
	to compile a 144,000 line program (it's an old program I'm playing
	around with):

	all : viola/viola

	libIMG/libIMG.a     : $(patsubst %.c,%.o,$(wildcard libIMG/*.c))
	libXPA/src/libxpa.a : $(patsubst %.c,%.o,$(wildcard libXPA/src/*.c))
	libStyle/libStyle.a : $(patsubst %.c,%.o,$(wildcard libStyle/*.c))
	libWWW/libWWW.a     : $(patsubst %.c,%.o,$(wildcard libWWW/*.c))
	viola/viola         : $(patsubst %.c,%.o,$(wildcard viola/*.c)) \
	                libIMG/libIMG.a         \
	                libXPA/src/libxpa.a     \
	                libStyle/libStyle.a     \
	                libWWW/libWWW.a

[14]	http://semver.org/

[15]	The whole intend of systemd is to enforce the *policy* of how a
	Linux system is booted.  Just do a search on "systemd sucks" for
	more than you wanted to know about "mechanism vs. policy".