[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index]
[Thread Index]
- Subject: An update on the LEM Project
- From: Sean Conner <sean@...>
- Date: Fri, 13 Jun 2014 22:46:21 -0400
A few weeks have gone by since I announced the project, which has some
mild support. I've been working on it since then, and I have made some
progress, or at the very least, gained some new information.
First off, I wrote the following modules [1]:
org.conman.zip
org.conman.zip.read
org.conman.zip.write
Those, with the addition of lzlib (and possibly with something like LFS or
org.conman.fsys) means we can read and write about 95% of all ZIP files.
There is some stuff I don't support (encryption, any of the documented file
extra data, or the 64-bit extensions) but for now, what I have works.
Now, the modules listed are low level routines, where knowledge of the ZIP
file format is required, but they're all written in straight Lua, with no
other dependencies since these modules don't bother with the
deflating/inflating or even reading/writing data. Nope, these modules just
deal with the ZIP file format.
I also wrote a lem module [4], which allows you to specify a LEM file and
it will load modules out of said file (or files---you can add multiple LEM
files). There are, however, still some issues.
First off, a sample LEM file:
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.env
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.errno
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.1 MODULES/org.conman.iconv
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.sys
SunOS sparcv9 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.syslog
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.base64
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.crc
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.env
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.0 MODULES/org.conman.errno
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.fsys.magic
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.hash
Linux x86 LGPL3+ Lua 5.1 5.1 1.1.1 MODULES/org.conman.iconv
Linux x86 LGPL3+ Lua 5.1 5.1 5.1 MODULES/org.conman.math
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.net.ipacl
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.pollset
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.process
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.strcore
Linux x86 LGPL3+ Lua 5.1 5.1 1.2.0 MODULES/org.conman.sys
Linux x86 LGPL3+ Lua 5.1 5.1 1.0.2 MODULES/org.conman.syslog
Linux x86 LGPL3+ Lua 5.1 5.1 MODULES/org.conman.tcc
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.cc
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.date
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.debug
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.dns.resolv
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.getopt
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.string
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.table
LGPL3+ Lua 5.1 5.1 MODULES/org.conman.unix
MIT Lua 5.1 5.1 0.10 MODULES/lpeg
MIT Lua 5.1 5.1 0.10 MODULES/re
Linux x86 MIT Lua 5.1 5.1 0.12 MODULES/lpeg
Linux x86 MIT Lua 5.2 5.2 0.12 MODULES/lpeg
MIT Lua 5.1 5.2 0.12 MODULES/re
Linux x86 MIT/X11 Lua 5.1 5.1 0.4.work3 MODULES/zlib
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket
Linux x86 MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.core
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.ftp
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.http
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.smtp
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.tp
MIT/X11 Lua 5.1 5.1 2.0.2 MODULES/socket.url
MIT/X11 Lua 5.1 5.1 1.0.1 MODULES/ltn12
FILES/APPNOTE.TXT
FILES/COPYING
FILES/README
FILES/Miscellaneous Things About Nothing
The Good: I still support multiple architectures, and modules written in Lua
can be loaded directly from the LEM file. Also, only those modules that
match the OS, CPU and Lua versions are loaded, otherwise, they're ignored.
I also include the version of the module, as well as the license.
The Bad: the modules written in C (so, a DLL or a shared object) *have* to
be written to disk before using. There is no way, short of writing my own
version of dlopen(), to support that. The limitiation is due, I suspect,
mainly due to "oh, we didn't think of that" with a minority of sysadmins
shouting from the wings "IT'S A @#$@#@#$@#$ SECURITY NIGHTMARE! DON'T ALLOW
IT!" [2]. So don't exepect that to change any time soon (Or ever).
The Ugly: I ended up monkey patching loadfile() and dofile(). That came up
as I was packaging LuaRocks into a LEM file:
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.add
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.admin_remove
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.builtin
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.cmake
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.command
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.build.make
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.cache
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.cfg
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.command_line
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.deps
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.dir
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.doc
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.download
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.cvs
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.git
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.git_file
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.hg
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.sscm
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fetch.svn
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.lua
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.unix
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.unix.tools
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.win32
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.fs.win32.tools
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.help
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.index
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.install
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.lint
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.list
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.loader
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.make
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.make_manifest
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.manif
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.manif_core
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.new_version
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.pack
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.path
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.persist
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.purge
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.refresh_cache
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.remove
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.repos
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.require
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.search
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.show
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.site_config
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.patch
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.tar
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.tools.zip
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.type_check
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.unpack
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.util
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.validate
MIT/X11 Lua 5.1 5.2 2.1.2 MODULES/luarocks.write_rockspec
MIT/X11 Lua 5.1 5.2 2.1.2 APP/luarocks-admin
MIT/X11 Lua 5.1 5.2 2.1.2 APP/luarocks
Lua 5.1 5.1 SCRIPTS/config-5.1.lua
Lua 5.1 5.2 SCRIPTS/config.lua
LuaRocks looks for config-VERSION.lua and config.lua, and to get around
that, I had to do the aforementioned monkeypatching. Well, had dofile() not
called the C version of loadfile(), I could have minimized the monkey
patching to just loadfile(), but seeing how dofile() uses the C API, I also
needed to override that function as well.
But I'm lucky in that LuaRocks only reads these files, and can run quite
nicely from the LEM file [3]:
[spc]lucy:~/projects/LEM>./tstapp.lua search zlib
Search results:
===============
Rockspecs and source rocks:
---------------------------
lzlib
0.4.work3-1 (rockspec) - http://rocks.moonscript.org
0.4.work3-1 (src) - http://rocks.moonscript.org
0.4.work3-1 (rockspec) - http://luarocks.org/repositories/rocks
0.4.work3-1 (src) - http://luarocks.org/repositories/rocks
0.4-1 (rockspec) - http://rocks.moonscript.org
0.4-1 (src) - http://rocks.moonscript.org
0.4-1 (rockspec) - http://luarocks.org/repositories/rocks
0.4-1 (src) - http://luarocks.org/repositories/rocks
0.3-3 (rockspec) - http://rocks.moonscript.org
0.3-3 (src) - http://rocks.moonscript.org
0.3-3 (rockspec) - http://luarocks.org/repositories/rocks
0.3-3 (src) - http://luarocks.org/repositories/rocks
0.3-2 (rockspec) - http://rocks.moonscript.org
0.3-2 (src) - http://rocks.moonscript.org
0.3-2 (rockspec) - http://luarocks.org/repositories/rocks
0.3-2 (src) - http://luarocks.org/repositories/rocks
0.3-1 (rockspec) - http://rocks.moonscript.org
0.3-1 (src) - http://rocks.moonscript.org
0.3-1 (rockspec) - http://luarocks.org/repositories/rocks
0.3-1 (src) - http://luarocks.org/repositories/rocks
Had LuaRocks written to any of the files, then what? And that's where I'm
stuck right now. Running Lua modules directly (for various values of
"direct") works, but trying to run a Lua application from a LEM file might
be asking too much, especially if there are persistent files. As a
distribution format, it's not bad, and perhaps if there was a command to
extract the various modules, scripts and files and write them to appropriate
places that would be the direction to go in.
One other thing to mention: the meta data, the OS, CPU, license, versions
and what not, aren't stored in a separate file, but are, instead, stored
using an extention method described in the ZIP file format. This extra data
isn't Lua specific and it could be used for other scripting languages. The
fields of this extended file data are:
version - version number of the file
license - license
language - language (perhaps application, like WoW?)
lvmin - minimum version of the language required
lvmax - maximum version of the language required
cpu - architecture required (for C-based modules)
os - operating system required
osver - operating system version
The fields are free-form strings, not longer than 255 characters. You
could easily throw in scripts in other languages, and while I didn't start
out with a universal scripting language application distribution format,
there's nothing in the format that restricts it to just Lua.
-spc (As always, comments and questions are welcome ... )
[1] No rockspec yet, as this is still somewhat in flux, but
https://github.com/spc476/lua-conmanorg/blob/master/lua/zip.lua
https://github.com/spc476/lua-conmanorg/blob/master/lua/zip/read.lua
https://github.com/spc476/lua-conmanorg/blob/master/lua/zip/write.lua
[2] To which I reply, "the computer is running. Game over."
[3] The code in question:
lem = require "lem"
-- ******************************************************
-- Any module loaded past this point *WILL* come from the
-- supplied LEM file. No ands, ifs or buts about it.
-- ******************************************************
package.path = ""
package.cpath = ""
local luarocks,err = lem("luarocks.lem","luarocks")
if not luarocks then
io.stderr:write("luarocks: ",err)
os.exit(1)
end
luarocks(unpack(arg))
[4] https://github.com/spc476/LEM/blob/master/lem.lua
