Re: Boxed userdata and garbage collector

[Mark Hamburg <mark@...>]

On Sep 3, 2010, at 1:29 PM, Roberto Ierusalimschy wrote:

>> See http://lua-users.org/lists/lua-l/2007-06/msg00088.html
>> 
>> Lightroom uses the call collectgarbage( 'step', x ) for some appropriate
>> value of x when working with images, but setting x is tricky. So, I
>> recently patched Lua as referenced above and have added an extraspace
>> field to our proxies together with an API to adjust the extra space
>> associated with the proxy. I haven't had things running long enough to
>> report on how well or even if it works, but it seems promising as a way
>> to keep Lua better informed of the memory in actual use.
>> 
>> It seems like this could be a good addition to 5.2.
> 
> As I explained in another message, it seems that keeping Lua better
> informed of the memory in actual use seems to have a negligible impact
> on the collector (except for making it run a little slower). Unless
> someone gives a good argument that this undestanding is wrong, I cannot
> see why it would be a good addition to 5.2.
> 
> In Lua 5.2, on the other hand, the 'x' in collectgarbage( 'step', x )
> will correspond exactly to the work the collector does when allocating
> 1KB of memory, so it should not be that tricky to set it.

The approach of calling 'step' when allocating the image (with an appropriate definition for 'step') seems to be conceptually equivalent to telling Lua about extra space in use which should drive the GC forward. What's lost is the ability to say, "but I manually released that space, so we can turn the pressure back down".

For example, say I've got code that generates a series of images. Sometimes those images are dispersed through the application, for example as previews, and will live on until no longer referenced and they get collected. (In fact, they may live in weak-table based caches.) At other times, they won't propagate and will get explicitly closed. This could happen when doing something like tracking a slider for an adjustment. Ideally, the code that built the image proxy should also tell the GC about the space in use by the image. But that then means that both use cases are treated identically and there is no way to get credit for the prompt closure of images. I could address this by replacing the close logic with a recycle path, but that both requires deeper plumbing to loop the recycled image back and potentially loses the benefit of having the closed image serve as an error indicator if references to the image live on (or we end up plumbing through the recycle path AND sending back a new image anyway).

Now, it may be that the way totalbytes works with the GC in 5.1 made this not really work as expected either, but on the surface it seems sensible. Memory pressure goes up and the GC should run faster. Memory pressure comes down and it should run more slowly.

Mark