Re: Incremental GC, step multiplier and finalizers

[Jean-Luc Jumpertz <jean-luc@...>]

Le 23 oct. 2013 à 16:28, Roberto Ierusalimschy <roberto@inf.puc-rio.br> a écrit :

> Sure. That "4" in the code was always magical. The problem is to decide
> what is the best approach. You already suggested two: make it change
> with stepmul or make it configurable. Maybe another option would be to
> make it dependend on the number of finalizers queued to run? (For
> intance, at each step run 5-10% of what is in the queue.)

Agreed. Running a fixed proportion of the queued finalizers could also be an option, but it seems to me that increasing this number with stepmul would also make sense in this case to avoid having too many finalizers to call when entering the gcpause state.

BTW I did more tests about garbage collection with one of my small test app on an iOS device to check the behavior of a same test pattern with different values of stepmul.

The test pattern is actually really simple: pushing and poping a single screen again and again (the screen being written in Lua). When the screen is pushed, its component objects are allocated, most of them having a native corresponding object allocated outside Lua, and when the screen is pushed those objects aren't referenced anymore and become candidate for being finalized during the next GC cycle.

Lua GC pause value being set to 125, two values of stepmul have been tried : 200 (the default) and 1600 (more aggressive).

The results are interesting: 

- with stepmul=200, the memory occupation of the program tends to grow to very large values in "big waves", with each memory usage wave growing higher than the previous one, despite the fact that the incremental GC cycles are regularly completed (i.e. reach the GCSpause state).
	E.g. after 8mn testing, the memory used grew at about 100 MB.

	Traces set in luaC_forcestep give patterns like:

	2013-10-29 14:49:30.852 Lua GC in state 2 called 4 finalizers
	2013-10-29 14:49:31.305 Lua GC in state 2 called 4 finalizers
	2013-10-29 14:49:31.530 Lua GC in state 2 called 4 finalizers
	2013-10-29 14:49:32.841 Lua GC in state 2 called 4 finalizers
	2013-10-29 14:49:33.065 Lua GC in state 2 called 4 finalizers
	2013-10-29 14:49:34.365 Lua GC in state 3 called 4 finalizers
	2013-10-29 14:49:34.681 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:34.893 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:36.215 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:36.447 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:37.657 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:38.085 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:38.311 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:39.476 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:39.706 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:40.976 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:41.335 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:41.558 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:42.718 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:43.226 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:44.493 Lua GC in state 4 called 4 finalizers
	2013-10-29 14:49:44.821 Lua GC in state 5 called 814 finalizers --> 15" in the sweep states

	2013-10-29 14:49:45.009 Lua GC steps: 92 --> number of times luaC_forcestep has been called since the previous GCSpause state

	2013-10-29 14:50:59.669 Lua GC in state 2 called 4 finalizers -> 1'15 in GCSpause and GCSpropagate state

- with stepmul=1600, the memory occupation stabilizes quickly at much lower levels (between 20 and 27MB) in much shorter waves. 

	A surprising result is that the number of steps in a GC cycle is smaller than in the stepmul=200 case, but not by factor related with the stepmul increment : 
		--> 67 steps when stepmul=1600 vs 92-130 steps when stepmul=200 !

	Traces set in luaC_forcestep give patterns like:

	2013-10-29 15:01:29.699 Lua GC in state 2 called 32 finalizers
	2013-10-29 15:01:29.903 Lua GC in state 2 called 32 finalizers
	2013-10-29 15:01:30.105 Lua GC in state 2 called 32 finalizers
	2013-10-29 15:01:30.307 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:30.509 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:30.717 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:30.919 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:31.121 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:31.321 Lua GC in state 4 called 32 finalizers
	2013-10-29 15:01:31.534 Lua GC in state 5 called 97 finalizers --> 2"  in the sweep states

	2013-10-29 15:01:31.730 Lua GC steps: 67 -> number of times luaC_forcestep has been called since the previous GCSpause state

	2013-10-29 15:01:49.708 Lua GC in state 2 called 32 finalizers --> 18"  in GCSpause and GCSpropagate state

- in both cases, calling collectgarbage() at the end of the test return the occupied memory down to 11 MB !!!

The complete traces and memory occupation graph can be downloaded at http://www.celedev.com/download/LuaGCTest.zip
The zip file also include a short screencast of the test to give a better idea of what it is about.

Therefore the question of how to improve the configuration of the Lua GC is probably larger than the finalizer distribution issue that was the subject of the initial message.

Would it make sense to add some "adaptive" capacity to  the incremental GC, so that it would increase the GC step size (whatever it means) when the memory tends to grow and avoid the kind of pseudo-memory-leak that I can see in the stepmul=200 case in my example ? (without running into a stop-the-world mode :)

Jean-Luc