Re: Ever-Increasing Memory Usge in 5.0

[Gavin Kistner <gavin@...>]

So, here's what I've found so far:

* Yes, it appears that we are using only 4-bytes in the userdata,
  with the rest of the information allocated in C++.

* However, if I never manually call collectgarbage() myself, and
  instead just spit out the information from gcinfo() frequently,
  no memory is ever freed. I have several pretty graphs, showing
  the count climbing up linearly, and then the threshold nearly
  doubles (actually, the numbers show threshold = ( threshold -
  54 ) * 2 almost every time).

  Oddly:
    * The threshold jumps well before the count reaches it. (For
      example, with a threshold around 24,000, it jumps to 48,000
      when the count reaches about 18,000.)

    * After the threshold jumps, the count also heavily jumps. With
      the same numbers as above, the count jumps from around 18,000
      to a little over 25,000 all at once.

* If I call GC manually TWICE every n frames, then I get a
  sawtooth pattern. This sawtooth starts out with significantly
  increasing memory usage/gcinfo threshold for the first 2-3 GC
  hits, but after that the memory usage (and gcinfo() threshold)
  levels out, peaking and dropping to the same values.

  Values of 'n' here are 100-40,000. When the program starts up,
  it's at 11MB. The first double-GC brings it down to around
  16MB, the next to around 20MB, and after that it flattens out,
  reclaiming all memory to the new 'low' of 22MB or so. The only
  guess anyone here could make is that memory is being
  irrevocably lost due to crossing paging boundaries, or
  something. This would also explain why GC every frame or so
  keeps the memory usage low.

* If I call GC manually ONCE every n frames (in the graphed case,
  2,000), memory is mostly never reclaimed, but eventually
  flattens out. The app starts at 11MB, and keeps climbing, until
  it plateaus at 80MB. There are a few short-lived plateaus along
  the way, and one short drop at one point, but mostly it just
  climbs.

* If I allocate a lot of temporary lua-only objects (tables,
  strings) each frame, I never see any climb in memory over time,
  even without calling GC manually.

I understand what Roberto wrote about a double-GC being needed
for C++ memory with __gc metamethods to be freed, but it looks
like that memory is sort of lost if the 2nd GC isn't right on
the heels of the first.

I've uploaded the graphs of what I measured to
http://phrogz.net/AnarkSamples/LuaGC/
because picturing what I describe above is easier as a picture.


So, my current questions:

1) Why is Lua's own GC not able to keep memory down on its own,
   even when its count hits the threshold?

2) Why does Lua's threshold double before the count reaches it,
   and why does the count massively increase then, too?

3) In the cases where I was calling gcinfo() every frame and
   every other frame, and writing the information to a file, you
   can see that the numbers varied far more than when I called it
   every 4, 10, or 100 frames. Why does the count suddenly drop
   and then rise again? Did Lua forget about some allocations, and
   find them again later?

4) Why do the first manual calls to GC not free up all the
   memory? Is that really just memory fragmentation? If so, why
   does it take so long to stabilize, and why does it stabilize at
   all?

5) Why does calling GC once on frame 1,000 and then again on frame
   2,000 not allow the C++ memory to be freed?
   
   
Thanks for any continued help,

--
(-, /\ \/ / /\/