Re: Lua Distributions and Package Management

[Peter Drahoš <drahosp@...>]

On 26 Sep, 2013, at 20:04 , Paul K <paulclinger@yahoo.com> wrote:

>> The wxLua editor is cool, but have a look at Paul K's Zerobrane
> Studio, which is the souped-up and generally refined descendent of
> that editor, reborn as an IDE.
> 
> FWIW, the next version of ZeroBrane Studio will package both LuaJIT
> and Lua5.2 and will work with both out of the box on all supported
> platforms (Windows XP+, OSX 10.6+, and various Linux flavors
> 32/64bit). It also includes its own lua.app (although it's hidden
> inside ZeroBraneStudio.app).
> 
> What I would like as a user is to have two commands: one would install
> the modules I need and the other one would put all the modules I need
> for a script in one folder (so that the script and all its
> dependencies can be copied to a different computer).
> 
> The first command would require a cross between LuaRocks and LuaDist
> to have a repository of modules to query with references to binaries
> for various platforms. I think "dist" from LuaDist comes close, but I
> don't see it working with binaries (this is running from LuaJIT):

Actually it does work with binaries. There are some minor issues that need to be ironed out because dependency resolver in LuaDist does not take into account ABI incompatibility that emerges once the module has been build with certain dependencies. This is probably the only issue that prevents LuaDist to fully distribute everything from binaries, even for luajit and lua-5.2.

> 
>> ld = require 'dist'
>> ld.install('lpeg')
> Downloading repository information...
> Finding out available versions of lpeg...
> Getting lpeg-0.11 (source)...
> Finding out available versions of lua...
> Getting lua-5.2.2 (source)...
> Error: command 'cmake' not found on system. See installation instructions at
> https://github.com/LuaDist/Repository/wiki/Installation-of-System-Dependencies
> 
> Note that it tried to get lua-5.2.2 even though I'm running from Lua
> 5.1 compatible system (_VERSION == "Lua 5.1"). Ideally, this would
> just get the binaries for lpeg and installs them where I need them to
> be.
> 
> Peter, could you make binaries available through ld.install?

This is an incorrect install of LuaDist. You are probably running LuaDist using an interpreter that was not installed using LuaDist itself thus it does have information about itself in share/luadist-git/dists directory that stores “dist.info” files about all installed packages, this is very similar to what LuaRocks does with rocks. Therefore the install function decided that it needed to pull in lua also. Note that it also decided to use source code which it does by default but setting lib/lua/dist/dist.config “source” setting to false will disable all build features and pull in only binary modules if available. 

This is what you need to do once you download LuaDist binaries from luadist.org (actual copy paste)
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> dist = require "dist"
> conf = require "dist.config"
> conf.source = false
> dist.install("lpeg")
Downloading repository information...
> lpeg = require "lpeg"
>

> I could then integrate this into ZBS to allow the user to type
> "require 'foo'" in the console and it would check that 'foo' is not
> locally available, would get its module description from
> LuaRock/LuaDist, grab the .lua file or the binary appropriate for the
> platform (and check the hash) and install it into the appropriate
> folder (probably under ZBS by default).

Michal Kottman already proposed and partially implemented a module that does exactly this based on LuaDist[1]. This is very much a proof of concept but it shows the possibilities.

> The second command would run your script with a patched version of
> "require" that would keep track of all the modules and their
> dependencies to see what needs to be packaged. It could then copy
> those dependencies into the project folder for distribution.
> 
> I know that binaries are not the best distribution format, but it does
> have some advantages over source code distribution; for example, I
> couldn't bootstrap luadist on my machine even though I have both
> recent mingw and mingw-tdm (it just kept crashing! on compiling
> luasocket 3.0). I don't have any numbers to support my claim, but will
> venture to say that the majority Lua users on Windows probably don't
> have cmake and mingw installed and will probably resort to searching
> for a binary on the Internet.

Agree, on Windows this is deep magic territory with many unknown variables and dragons nearby. That is why for windows a binary only distribution is essential.

> Another issue for Windows uses is that some libraries are compiled
> against different lua dlls. For example, ZBS libraries are compiled
> against lua51.dll and lua52.dll, but LuaDist libraries are compiled
> against liblua.dll, which makes them not to work with ZBS executables.
> I "fixed" this by adding a proxy DLL that forwards calls from
> liblua.dll to lib51.dll, but ideally we'd settle on one way to
> name/reference the libraries. I've considered something similar to
> liblua, but it has disadvantages for situations when one packages
> needs to support both Lua 5.1 and Lua 5.2 (as it is in my case), so I
> kept lua51 and lua52 names for dlls.

I agree that this is an issue. LuaDist “solves” this by using separate install folders for different interpreter versions. This was the most straightforward way to implement it on most platforms without issues. ZBS could simply contain separate subfolders for each interpreter.

In general LuaDist is “almost there” but needs some polishing and usability re-evaluation before it can be applied in the context you are looking for.

pd

[1] https://github.com/LuaDist/intellua