Re: Custom Lua Binary Sizes

[Paige DePol <lual@...>]

Paige DePol <lual@serfnet.org> wrote:

> As I've been hacking on Lua and adding more features this size has been
> increasing, with all my patches I have just hit a megabyte in size in a
> release build. 

Apparently, my Xcode was lying to me... repeatedly. It seems an old issue
I had with Xcode badly caching items and generally just being strange has
returned. After a judicious clearing of all Xcode caches and a recompile
of all object files in Release (-Os) configuration I get the following:

======== (15.5%)  (74.5%)  (34.8%) =======
 Parser:  35.4k +  82.4k = 117.8k (233.2%)
Runtime: 192.8k +  28.3k = 221.1k ( 14.7%)
Overall: 228.2k + 110.7k = 338.9k ( 48.5%)
======= Vanilla = Growth = Lunia =========

The top %'s are the size of the Parser in relation to the Overall size.
The side %'s are the growth of the binary code for that specific line.

My hard fork of Lua is actually only 48.5% larger than vanilla Lua at this
point, and about 3x smaller than previously indicated. As can be seen above
the bulk of my changes have occurred in code related to the Parser, with an
increase of 233%. The runtime code has only had an increase of 14.7%, which
is much more in-line with what I was expecting. If you remove the parser
from my fork the resulting binary actually winds up smaller than the vanilla
Lua binary (with parser, of course).

The size of the parser has significantly grown as it includes a number of my
larger patches: Preprocessor, Token Storage, CEMI (Consts, Enum, Macros, and
Inlines), Class Object Model, variable Type Locking, and more.

The runtime portion supports my new Index data object, which also required
a bump up to 64-bit for the virtual machine instructions, as well as support
for Type Locking of stack slots.

I have done my best to do as much processing as possible in the parsing phase
to allow the runtime to do as little as possible... so far I think that has
worked out quite well given the fairly small increase in the runtime code.


> Thomas Fletcher <thomas@cranksoftware.com> wrote:
>> For what it is worth, I would consider what you have packaged as being far too
>> large for us as we're running on systems with <1M RAM and then varying amounts
>> of flash. 

Yes, with only 1M RAM I'd imagine a Lua engine that was your entire memory space
would be deemed impractical. Thankfully, I discovered my numbers were off quite
a lot, with the new numbers how would that fit into your memory requirements?


> Sean Conner <sean@conman.org> wrote:
>>  For work, an application I wrote [1] embedded Lua plus a large number of
>> modules required for it to work.  The modules written in Lua are
>> pre-compiled; in some cases they are smaller than the source files; other
>> times not so.  In any case, these are further compressed using zlib before
>> being embedded in the executable. 

Wouldn't overall memory available be a concern as well with a compression
library? I am guessing for embedded systems the decompression happens as
a stream, so the only memory allocations are for the decompression engine
itself and the final decompressed file?


> Frank Kastenholz <fkastenholz@verizon.net> wrote:
>> I've not looked at your code, nor do I know the features you've added,
>> but in general, is it possible to partition your additions so that they
>> can be selected (via #define INCLUDE_FEATURE_X sorts of statement)?

Yes, this can be done, and it was a method I did use originally. However, I
found the extreme amount of #if statements unwieldy after a while, and
especially where various patches might change the same code, or overlapping
code areas. Instead, I will be managing patches individually, with pre-reqs
for some, and will maintain a build system to test my patches with all my
other patches... eventually this will lead to a website where you can have a
custom version of Lua created for you with just the patches you want.


> Sean Conner <sean@conman.org> wrote:
>>  I finally have some time, so I thought I might check the sizes of Lua
>> source code, compressed Lua source code [1], compiled Lua code and
>> compressed compiled Lua code, for both 32 bit and 64 bit systems...

Thanks for doing those comparisons, Sean, it was enlightening. Continuing
the later discussion about compression... I could see adding an option to
compress the debugging information in a compiled Lua file. That way the
debug info could take up less space and only be decompressed if actually
needed. Though if the error is a memory error that may prove problematic!


Thanks all for your feedback, it is appreciated. I am not yet playing with
embedded systems, however, I have a couple ideas so might be looking into
what hardware to get. Assuming I want to use Lua does anyone here have any
suggestions? Raspberry Pi, Arduino... I am a bit of a hardware newbie! ;)

~Paige