Re: Lua bytecode doesn't compress well (with LZMA)

[Matthew Wild <mwild1@...>]

2009/12/1 KHMan <keinhong@gmail.com>:
> Juris Kalnins wrote:
>>>
>>> Squish actually uses this as its main output filter,  (there's more
>>> than one, such as the one that replaces Lua keywords with single bytes
>>> > 128) - and it works really well.
>>
>> And it's very simple to modify llex.c to parse such alternative tokens,
>> so that reduced scripts can be used directly (btw, Lua also has 20+ unused
>> bytes from the 0..31 control character range).
>
> I have one query for Matthew, the author of squish. Want to find out (the
> lazy way) a few data points (always love data points) to get a feel of
> things:
>
> Does anyone have a set of data points for squish, where we compare
> data-compressed sizes of sources, where:
> (1) keyword token replacement filter, versus
> (2) no keyword token replacement
>
> Now, in (2), LZ coding would zap most keywords into a sliding dictionary
> match code, whereas in (1), the initial size of the filtered sources will be
> smaller but there is more variation in the symbol frequencies of the source
> code (token symbols added) and less chance to make sliding dictionary
> matches.
>
> So, would there be a big difference when we compare compressed sources? Say,
> we tabulate results as:
> (1) original
> (2) token filtered
> (3) original, compressed
> (4) token filtered, compressed
>

I'm working on a new release, and it will include these statistics (as
a pretty graph using my <another-plug>Google Charts API lib:
http://code.matthewwild.co.uk/gchart/ </another-plug> ;) ).

Matthew.