Re: Update on GC performance (github version)

On Fri, Nov 24, 2017 at 5:29 AM, Rodrigo Azevedo <rodrigoams@gmail.com> wrote:

short answer: if I have less than 892302.1 kbytes (lua 5.2/5.3) then that very simple script doesn't finish correctly. Is this really what you expect it to? (I don't)

There's a tradeoff between the amount of memory and the amount of CPU a program uses. Taking your first example again, but tweaking the GC parameters:

collectgarbage("setpause", 120)

collectgarbage("setstepmul", 800)

N = 2.0e7

C = {}

for i=1,N do C[i] = i end

print(string.format("%.1f",collectgarbage("count")))

for i=1,10*N do

local a = {1,2,3,4}

end

print(string.format("%.1f",collectgarbage("count")))

The GC now does not wait for the memory used to double before kicking in, and it does 4 times more work in each step.

The output (on a 64-bit amd-64 linux machine) is now:

524311.3

604036.1

The cpu time went up from 33.3 to 47.6 seconds. That's where the tradeoff comes in, you sacrifice memory for speed, or speed for memory.

long answer:

1) first 'for' means: a lot of work for GC manage (dozens Gbytes), consuming almost all we have of free memory.
2) second 'for' means: perform a lot of complex tasks for many hours/days that also create a lot of small (~ Kbytes) GCed temporary objects at a high rate.
3) go to the short answer.

Yes, that is a "toy model" but I also observed the same trends testing with some 'real' (long answer) programs, so that is a valid model.

Observe the attached files for an updated teste2.lua program and also the resulting graph showing the memory usage for lua-5.2.4 generational GC, lua-5.3.4 and github codes. What did you expect it to?

A generational collector make a different tradeoff, objects that have been allocated for a long time will not be marked by the collector in every pass. This is helpful for programs that create lots of short-lived objects, which is the behavior you observed. However, if you run your example program twice within the same VM 'C' will be recreated, but the old copy will not be collected immediately, instantly doubling memory use.

There is no magic bullet, automatic memory management comes at a cost in both memory and CPU use.

Gé