Re: [ANN] LuaBinaries

[Mike Pall <mikelu-0503@...>]

Hi,

Danilo Tuler wrote:
> But maybe this is just for historical reasons.

It's nice to have a separation in the Makefile between the core and
the library files for embedding Lua in restricted environments.

However the moment we are talking about shared libraries, this point
becomes moot. Primarily because shared libraries imply a virtual
memory system, a CPU with an MMU, page size granularity, relocation
tables and so on ... The size of the Lua core and Lua core library
is really insignificant then (have a look at the size of libc).

A single shared library makes much more sense (in the good Lua tradition
to keep things simple). VM systems only map in those code pages that
are actively in use, anyway. This is not going to get better by splitting
things up into two libraries (in fact it's likely to be worse).

But read below why I think that one should not put the Lua core into
a shared library at all (if possible).

> Poll: does anyone do not link with liblua? Do you always use both libraries?

I doubt that anyone does this except in a pure embedded environment
(but shared libraries are unusual there). And lauxlib is in liblua
which is another reason to link with both.


On POSIX systems with ELF (or similar) binary formats I compile
everything into a single 'lua' executable that is made with -Wl,-E
(or equivalents). Loadable modules depend on symbols from the
executable.

On Windows I compile everything into a single lua51.dll except for lua.c
which goes into lua.exe (linked to lua51.dll). Loadable modules depend
on symbols from lua51.dll.


I strongly recommend *NOT* to provide shared libraries on systems
where this can be avoided (that is everywhere except for Windows).
I've previously explained the reasons and I guess (I hope) this is
why the shared library target is gone from the Lua 5.1 Makefiles
(please don't resurrect it as part of a binary distribution).

There are generally two ways one can use the Lua core: 'embedding'
the core into your own executable or 'extending' the Lua standalone
executable (or some variant of it) with your own modules:

- Embedders couldn't care less about premade libraries. Neither do
  game developers. They will compile their own (probably heavily patched)
  sources. Don't worry about them.

- Anyone extending Lua is using the 'lua' executable only (with the
  core statically linked). On any reasonable virtual memory systems
  this executable _is already shared_ in memory. There is no point
  in putting the code into a shared library just for that purpose.

- The Lua core is so small that using shared libraries would not
  noticeably improve memory sharing or disk space sharing, anyway
  (at least when we are talking about the category of systems that
  have virtual memory).
  Thus anyone that has the desire to replace the 'lua' standalone with
  its own variant should do so by statically linking in the whole core.
  However with the new module system in place, I can hardly see a point
  in doing so (it's easier to use the provided executable and just load
  some modules).

- Putting the code into a shared library comes with a hefty performance
  penalty on x86 when compiled with -fPIC. This steals one whole register
  (EBX) of 6 available general purpose registers. Although some x86
  ABIs permit compilation of shared libraries without -fPIC, this is
  against the release policies of several distributions. The penalty
  (one relocated copy for every process) is minor in the case of Lua
  since it's so small and a multi-process scenario is unlikely
  (a threaded or non-preemptive scenario does not suffer from this).
  However those distributions won't change their policies because of us.
  Since x86 is the predominant CPU architecture, we really want to
  avoid the shared library performance penalty.

- The advantage of smoother upgrades with shared libraries is not
  relevant for Lua since it is explicitly stated that there is no
  binary compatibility, even amongst minor versions. E.g. you need
  to recompile everything when moving from Lua 5.0 to Lua 5.1.

Case in point #1: ever wondered why the 'perl' executable is almost the
same size as libperl.so? Simple: the executable is not linked against
the shared library! It contains the same code, just statically linked.

This is precisely because of the above reasons. In fact everything
is compiled twice (once with -fPIC and once without) to get the most
speed out of the standalone executable.

Case in point #2: at least one major distribution builds Lua as a shared
library. And guess what -- this distribution is used by the 'Great
Computer Language Shootout' and that's one reason why Lua scores lower
on it than one might hope for.


Oh and BTW ... about the list of systems the binaries are built for:
I doubt the Linux kernel version is relevant. What really matters is
the libc version. And, err ... Motif? This is not present on most modern
distributions (unless you explicitly install it) and the Lua core doesn't
link with it. I guess this is for IUP?


Anyway, I welcome the effort to standardize on some binary distribution
conventions for Lua. This is much needed. Thank you for your work!

Bye,
     Mike