Re: Feature request: more integration of Lua and C memory management

[Jamie Webb <j@...>]

On Thursday 10 June 2004 01:39, Bilyk, Alex wrote:
> Well, the post you are referring to outlines some real problems. However,
> even though the argument is made about Lua deficiency in a way, I view the
> post as describing problems that are application specific.

Except that I'm not talking about an application, but a library. The idea is 
that a lot of applications will be created using this library. The current 
choice is either to make the library slow and perhaps troublesome to 
maintain, or to make it place specific, and fairly unreasonable, restrictions 
on Lua code which uses the library (viz. don't create cycles).

> Clearly the
> problems descripbed will emerge in any application that aggregates objects.

Not at all. Failure to collect reference cycles is a problem associated with 
specific memory management strategies.

> If I have two C++ objects A and B that reference each other and both of
> them are refcounted I have the same sort of problem.

Certainly. That's one reason not to use reference counting. But, if I code in 
C++ and decide to do reference counting then I should also expect to take 
reasonable steps to prevent cycles. If I code in Lua OTOH, I shouldn't have 
to worry about that sort of thing.

> Such problems don't
> belong to programming langage domain in my view (although .NET does
> something in this department but only via modifications to C++ language).
> Wtih Lua we have the same thing -- Lua references its objects in one way
> and userdata does (or doesn't) the same in some other way for what is
> stored inside it. This amounts to two referencing systems for two different
> kinds of objects that happen to relate with each other in some way.

The trouble is that 'some way' introduces memory management overlaps which are 
a pain to resolve correctly, and that fact is a consequence of the language 
design.

> I think, no matter what we do one problem will persist and that is:
>
> 	When the last Lua reference to a userdata object is gone there is no
> guaranty that C object stored in such userdata has no outstanding
> references on the host app side and vice versa.

Not unless the C code provides a such a guarantee, and it /should/ provide 
one. The problem is that it is not presently easy to do this in the general 
case without introducing the possibility of uncollectable reference cycles.

> Pooling userdata is a problem that has no clean solution in the existing
> Lua implementation. This could pbobably be fixed by allowing a special
> userdata allocator routines to be defined by the host application or just
> making existing allocation routines take another parameter describing the
> type of a Lua object being allocated, so that custom memory managers can
> take a note of it and use pooling when needed. A fixed size custom
> allocator is another way to solve this problem. The latter in effect would
> pool everything allocated by Lua (ala STL in C++), being 10x faster while a
> bit more wasteful as a side effect.

The pooling thing wasn't a very well-formed idea. I suspect that the time 
saved by pooling memory allocations would be dwarfed by the use of GC to 
clean them up. I was talking more about pooling of live objects such as 
database handles, where the GC is used to determine when a handle is returned 
to the pool, but of course that can be done fairly easily by decoupling the 
Lua-side and C-side handles.

-- Jamie Webb