Re: Strange memory allocation behavior

[Luke Horgan <horgan.l@...>]

Hello all,

Thank you so much for the information you've provided!  Tom, your
answer in particular has answered many questions I had.

I have not disabled the garbage collector for performance reasons.
The issue (or part of it) is that the contract scripts are supposed to
be strictly deterministic, and I wasn't sure if there was any
randomness or unpredictability in the way the garbage collector
chooses when to run, or what to collect.  Every script has to have
exactly the same output on every peer's machine.  You can imagine a
scenario where someone deliberately writes a script that creeps up on
the memory limit.  If the garbage collector isn't totally consistent,
then maybe it swoops in and frees memory on one user's machine before
it has a chance to fail, while on another user's machine it doesn't.
If I keep a running tally of total allocated memory, will that number
be consistent on every run, on every machine?  I suppose I could
experiment a bit to find out, although it would be nice to gain some
insight into how the collector actually works behind the scenes.

Best,
~Luke
On Mon, Sep 17, 2018 at 7:28 AM Tom Sutcliffe <tomsci@me.com> wrote:
>
> On 14 Sep 2018, at 5:35 pm, Luke Horgan <horgan.l@husky.neu.edu> wrote:
> >
> > Before running a contract script, we disable
> > the garbage collector with 'collectgarbage("stop")'.  Hypothetically,
> > this shouldn't be a problem, since we only want scripts to be able to
> > allocate a fixed amount of memory, total, over the entire course of
> > their execution.  However, it seems to cause the memory profile of
> > even trivial scripts to blow up for no particularly apparent reason.
> >
> > We are tracking allocations using a custom lua_Alloc function, as
> > documented in the reference manual (version 5.3).  Of particular
> > confusion is that functions which are never called still appear to
> > consume large amounts of memory (hundreds of kilobytes).  This problem
> > does seem to be severely exacerbated by the use of custom variables we
> > load from the sandbox environment, so I can only imagine that's part
> > of the problem, but the behavior is confusing regardless.  Why should
> > an uncalled function consume so much memory, or any memory at all?  Is
> > there any reason lua_Alloc might behave strangely with the garbage
> > collector disabled?
>
> I doubt it is anything specifically to do with lua_Alloc or uncalled functions, it might just be you're seeing more of the internals of how Lua works. Every scrap of memory Lua uses internally goes through the alloc function, not just the memory used for the variables and data structures your script declares, and there can be quite a short-lived objects created.
>
> For example, are you parsing Lua source code while the collector is stopped? If so then any temporary memory needed by the parser's internal data structures will not be freed up either.
>
> Also, parsing Lua source itself requires memory to store the bytecode implementation of the functions. So the more stuff you parse, the more memory it'll consume even if you never call it. As a _very_ rough rule of thumb, the bytecode tends to take up slightly more memory than the original source code (although I don't remember any concrete figures).
>
> I'm not sure I understand the perceived value of running the contract logic with the garbage collector disabled, the language by its nature uses the allocator a _lot_ behind the scenes, and running for prolonged periods without the collector ain't gonna be pretty. The really nice feature of an incremental collector is how the penalty of collection is spread out so there aren't any unpleasant stop-and-halt delays. The only time I've ever considered disabling a garbage collector was in languages which didn't have an incremental collector and I was doing something time-critical for a short period. I would never bother in a Lua script - generally the only time I ever call collectgarbage() is to force collection of native objects which need _gc-ing, or for debugging, or when I'm not going to use the lua_State for a while and want to minimise its memory footprint.
>
> If you want to impose a maximum allocation limit on the scripts, I'd suggest you allow the GC to run as normal, but keep track of the currently-allocated total in your lua_Alloc function rather than the cumulative sum of all allocations. As 云风 mentioned, returning NULL from the allocator generally triggers a full garbage collection cycle anyway which gives the runtime a chance to GC stuff and get back under budget again, so there isn't a huge amount you need to do to make it work.
>
> I've had some fun in the past running instruction count and memory constrained Lua interpreters, so it's definitely something the language can handle, providing your native functions aren't too much of a CPU-usage multiplier, as the hooks can't really help with "while true do reallySlowNativeFunction() end". Just let the garbage collector do its thing :-)
>
> Cheers,
>
> Tom
>
>


-- 
Luke Horgan
Candidate for BS in Computer Science and Computer Engineering
College of Engineering | Northeastern University
luke-horgan.com | 908-577-7902