Re: Lua bytecode doesn't compress well (with LZMA)

[KHMan <keinhong@...>]

Matthew Wild wrote:
> 2009/11/30 David Given <dg@cowlark.com>:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Luiz Henrique de Figueiredo wrote:
> > > > Shameless promotion: http://matthewwild.co.uk/projects/squish
> > > Another shameless plug: http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lstrip
> > Just because one can never have too many types of wheel, there's also
> > Kein-Hong Man's luasrcdiet, which I'm using very successfully in Prime
> > Mover:
> >
> > http://luaforge.net/projects/luasrcdiet/
>
> Squish actually uses this as its main output filter,  (there's more
> than one, such as the one that replaces Lua keywords with single bytes
> > 128) - and it works really well. I went a step further because I
> wanted to bundle multiple libraries as well as other files (non-code)
> into a nice compact file.

That aside, I just checked a vanilla 87KB script compiled -s to 
44KB, and 32% of the symbols were zeros in the binary chunk. 
Switch those many 32-bit count values into a variable number of 
bytes (the first byte can range from 0-127, etc.), and the binary 
chunk can be made a little smaller. Little loss of performance, no 
variable-bit encoding schemes, but not nearly as good as proper 
data compression (say, a 25% improvement is not really much vs 
LZ+statistical coding schemes).

Another way (not very practical, and probably a patent minefield 
to boot) would be to encode bytecode in reduced form, like the ARM 
Thumb instruction set. Register indices in Lua bytecode probably 
very rarely use their full dynamic range, so one can design a more 
compact instruction set encoding that is still byte aligned and 
save some bytes. More of an academic exercise, though.

The nice thing about standard bytecode is that it can be 
'compiled' by Lua roughly as fast as it can be read from a hard 
disk. On a contemporary desktop PC, it's hard to justify size 
reduction contortions... It can be an option for certain specific 
end-use areas of a piece of software.

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia