Re: Ever-Increasing Memory Usage in 5.0

[Gavin Kistner <gavin@...>]

On Apr 11, 2006, at 3:47 AM, Daniel Silverstone wrote:
> My guess is that you're boxing the matrix pointer in a userdata,  
> and as
> such, your userdata is four bytes big. This is all Lua takes into
> account so I'm guessing your sawtooth will be present, just incredibly
> large.

I'll verify your guess today, but let's assume that it's true, as  
it's a "worst case" scenario for the discrepancy between Lua's size  
and the C++ memory side. Let's assume that a Matrix is 80 bytes in C+ 
+, and 4 bytes in Lua. (I don't really know, but it's at least 16x4  
bytes of data, and we'll throw in some overhead to be sure.) As I  
mentioned before, the GC threshold starts out at 82k.

Absent any calls to GC, that means we can have roughly 21,000  
Matrices hanging around in Lua before the threshold will be hit. At  
80 bytes per, that's 1.6MB of memory waiting on the C++ side of  
things. At this particular threshold, I should see a sawtooth pattern  
on the order of a couple megabytes. However, I've watched the linear  
memory increase go for at least 5 megabytes.

Further, when I call collectgarbage() the first time, that's an  
implicit parameter of 0, forcing garbage collection. According to the  
5.0 documentation, the garbage collection should run (leaving a  
single Matrix around, and perhaps a few other Lua objects in this  
test), update the count, and set the new threshold to twice the  
count. At this point, my threshold should be very small, and (if the  
auto-GC was working) I should see a much smaller sawtooth. But, a  
single call to collectgarbage() does not induce any sawtooth later  
on, again as memory grows by multiple megabytes.


> Simple memory fragmentation at a guess. At points less regularly than
> once per frame you are allocating an object which is longer lived than
> the per-frame objects.

Unless we've missed something, this particular guess shouldn't be the  
case. We have internal memory trackers, and (separate from this  
issue) the simulation is running at a steady state, with 0 memory  
allocations after startup.

(Hrm...though Heisenberg may be smacking us upside the head. The act  
of flipping the gc-now flag was being done through a scriptlet  
mechanism that does use memory allocations. I'll have to try a pure- 
code flipper and see how that goes. Thanks for the idea.)



> When I last worked on a game engine, we would force a garbage-collect
> once per second which was less irritating than every frame but often
> enough to keep memory use down. But we weren't using Lua so I don't  
> know
> how Lua would behave in this instance.
>
> Remember that as Thomas said, the size of the userdata from the  
> point of
> view of the garbage collector is the number of bytes you allocated  
> (4 or
> 8 for a boxed pointer) and the size of the Udata header which is
> probably somewhere in the region of 16 or 20 bytes on most common  
> 32bit
> architectures.

Thanks so much for all your help. This is, in fact, for a game engine  
(sort of). As I've noted, I'll get stronger numbers today as to where  
memory is stored, what the gcinfo() threshold and usage is reporting  
during the increase, and so on.