Re: Lua idea

[Jay Carlson <nop@...>]

On Mon, Apr 9, 2012 at 8:28 AM, steve donovan <steve.j.donovan@gmail.com> wrote:
> On Mon, Apr 9, 2012 at 12:19 PM, Luiz Henrique de Figueiredo
> <lhf@tecgraf.puc-rio.br> wrote:
>> Only if you glue scripts *after* upx'ing them.
>
> OK, so only the base executable is compressed?  That's fine -
> personally I think it's easy to go overboard with size optimizations,
> which curiously enough are _not_ memory optimizations.  Big fat 2 MB
> executable without significant external dependencies launches faster
> and uses less memory than 100 KB executable with dependencies; zipping
> is good of course for delivering them.

(I know most of the people in this thread already know this, but the
arguments are not going to make much sense without some background.)

A memory management koan, to break the rational mind--or at least the
rational mind of people who grew up programming on personal computers.

You have a Linux process on, say, MIPS. How can you find out how much
memory that program is using?

Answer: It's trivial. The program is using 4K (the memory page of the
current instruction) or 8K (the target page if the instruction is also
doing a load or store).

Every other page may not be in memory. Read-only pages, such as const
data or instructions, may be dropped by the kernel at any time and
reloaded from disk. Copy-on-write pages, such as initialized data, may
be dropped until they are modified; after they're modified they may be
written to swap; if they're in swap they may then be dropped at any
point as well.

Dynamically allocated memory may be thought of as copy-on-write of /dev/zero.

We do talk about "this process is using 200k of memory" but this is
confusing shorthand for dynamic behavior: if those 200kbytes (either
read-only or read-write) are dropped by the kernel, they will soon be
accessed again, and the process will eventually become slow enough
that we kill it in despair. Usually we're talking about "hot" pages;
code paths never touched don't matter; those pages only run at startup
are cheaply discarded. Since memory usage is a characteristic of
behavior, we can also talk about ranges of memory usage usefully.

Standard executable packing techniques dynamically allocate all the
space for the code. This still doesn't mean that a UPX'd MIPS
executable is using more than 8k of code. But it does mean that all of
its code pages have to go to swap before they can be dropped; these
pages include the startup and dead code.

Machines without swap...then have a problem with writable memory. On
machines that small most people also use a compressed filesystem like
squashfs. Unlike a UPX'd executable, the kernel does know it can
always reload a read-only page, and therefore can drop them without
having to put them into swap. It might be interesting to write
something like UPX in userspace that would somehow let pages be
dropped and then trap segfaults to page things back in. I forget if
you can do that outside research OSs....

Anyway, things like UPX or executables with DLLs/.so files in trailing
zips work OK if they're run one-at-a-time or infrequently. Running
multiple copies at once ends up being significantly more expensive
than it looks. For Lua source/bytecode, UPX is not good since the
pages with the source are dead once load()ed into the state; at least
with a trailing ZIP file, the kernel is not obliged to keep a writable
(but never again written) copy of their portion of the text segment in
swap.

Practical considerations trump theoretical optimization. If something
isn't run that often, and it's run on a memory-rich machine, you can
probably figure out it doesn't really have perceptible overhead--the
idiom "no harm, no foul" is appropriate. And it doesn't help to
optimize a package for lots of use if *nobody uses it* because it
takes up too much disk space or it's too complicated to get all its
library/module dependencies installed etc etc etc.

Jay