Re: Memory leak?

["Mark Meijer" <meijer78@...>]

Thanks all for your replies and insights. Never knew about the
remembering keys thing, but it makes sense. See below for individual
replies.



Stephen Kellett wrote:
> http://www.softwareverify.com/lua/memory/index.html

Thanks, looks like a useful tool.



Patrick Donnelly wrote:
> I'm pretty sure this has to do with tables remembering "old" keys.

Mike Pall wrote:
> You are setting the values to nil. The keys stay there. In this
> case the keys are implicit, since it's all stored in the array
> part. But the same statement holds for (say) string keys.

Hans Hagen wrote:
> so, in your case, once you have a t[10] defined you can set it's value
> to nil, but that does not dispose the variable, only its value

Good points. I just figured it would all simply be garbage collected.
To make a bit more clear, in the following example, is what the string
says correct?

foo = {}
bar = {"This string is never cleaned up."};
foo[bar] = nil;
bar = nil;
while(true) do end



Patrick Donnelly wrote:
> This is important for next() when you try to check the next key/value
> pair 'after' removing the last... e.g.
> for k in pairs(t) do t[k] = nil end

Mike Pall wrote:
> [Other languages don't allow you to modify containers at all
> during traversal. Pick your poison.]

Indeed, I was a bit surprised to see that this is valid in Lua. I
kinda like it. I always assumed that Lua allows it only with ipairs().



Alex Mania wrote:
> But note that this will also fix the problem:
>  -- So far so good... But now spot the difference...
>  print("Around 16meg too much:", GetUsedMemory());
>  foo.hashpart = true   -- no need to reassign foo to a new table)
>  print("Back where we started:", GetUsedMemory());
>
> The only reason it shrunk the array when the hash table was used in the example
> was because the hash table had previously been unused - requiring it to grow and
> reevaluate its array size. So it looks like you can't count on a table being
> shrunk...

Great :) At least there's a way around the problem. Building on the
above, I have found that "foo.bar = nil" will consistently and
repeatedly reclaim the memory used by "remembered" keys. Apparently,
when assigning it nil instead of some other value, the "bar" key is
reclaimed immediately as well, which allows the same trick to work
again and again.



Alex Mania wrote:
> I doubt that is the problem in your program though. (No memory is leaked - it's
> always reused..)

That's true it's reused, but my stress test involved a million
coroutines that each have their own message queue (which is initially
empty). Then the main thread starts randomly sending messages around.
That's basically the test. Now, as more and more of the threads start
getting messages, more and more keys will be "remembered" in their
respective message queues, even after the messages have been spooned
out again. And even more so, if they ever have more than a single
message in the queue, which is likely. I realize this is an extreme
case, but I still think I know why it runs out of memory when running
long enough :)



Mike Pall wrote:
> Yeah, I know what comes next: you'll complain that this behaviour
> is unexpected. Maybe it is, but there's no easy way to fix this
> and still be able to modify values for existing keys during table
> traversals (e.g. set them to nil) and at the same time keep the
> garbage collector happy.

Alex Mania wrote:
> I have a feeling any possible patch would slow table setting considerably =/

It is a bit unexpected :P but I'm not complaining about that really
(although it probably should be documented somewhere).

But I do think there needs to be some way to affect this. I'm not
saying this behaviour should be changed at the cost of certain
features, or at the cost of efficiency (please don't!). But at the
very least clean it up when the garbage collector is explicitly called
:P and maybe add something to the list of allowed arguments to
collectgarbage() which does just that. Say, collectgarbage("keys"), or
something.

Right now it's all behind closed curtains and there's nothing anyone
can do to reclaim that memory. Well, unless of course you happen to
know how to force the cleanup. So all I'm saying is there should be a
way to do it that is formally supported by the language.



Mike Pall wrote:
> Tables expand automatically, but shrink only under certain
> implementation-specific conditions. Under Lua 5.x this happens
> only when resizing a table. This is triggered by inserting a key
> which neither fits in the array nor the hash part.

Does that mean that Alex's trick above will only work sometimes,
depending on what keys are already in the hash part?



Mike Pall wrote:
> Lua does not optimize for your use case because it's not good
> programming practice, anyway. You'd usually just drop the
> reference to the table (and get a new one) instead of clearing
> all values. It's a lot faster, too.

Alex Mania wrote:
> I would be very interested to see a program in which this would be a
> problem though. Most high speed allocators rarely shrink blocks anyway; even if
> requested.

Obviously I didn't know about the conditions which cause that stuff to
be cleaned up. I'm sure some thought went into that, because if it'd
never get cleaned up at all then I'd say it would be a big problem.
For any app that gets to see a lot of different keys in their
lifetime, and/or usually runs for long periods of time. For example
anything that does a lot of reverse-lookup/indexing, or caching or
such things. The Kepler Project might not be possible. But yeah, the
way it is, I guess it's not a major issue :)


Thanks all!

Mark