Re: GC [was Re: Lua 5.1 (work4) now available]

[Mark Hamburg <mhamburg@...>]

The caveat for the following analysis and recommendations is that I feel
distinctly sleep deprived right now...

I'm perusing the 5.1work3 sources since that's what I'm running right now. I
think I can explain Mike's graph:

* The red upward slopes represent the time when the GC does nothing because
the step logic is disabled while totalbytes is less than GCthreshold.

* Nursery collections tenure everything that survives and the new total is
used to set the new value of GCthreshold. That's why the estimate and
GCthreshold tick up in reasonably regular steps. That's the new data being
tenured.

* We have a delay in triggering a full collection. We trigger a full
collection on the next round if the value of estimate is twice its value for
the previous full collection. So, the baseline usage is somewhat under 1000
on Mike's graph. We can assume that prevestimate got set to this number.
Estimate eventually rises above 2000 at which point the cycle after that one
-- where estimate will be nearly 3000 -- is set to be a full GC cycle.

* Since estimate is nearly 3000 on this cycle, GCthreshold is nearly 6000.
Furthermore, since the full collection is incremental, we will keep
allocating storage for a while past the threshold while we drive the
collector forward. That's the little upward slope past GCthreshold before
the big plunge.

The net result is with a baseline state just under 1000, this manages to
drive totalbytes up to nearly 6000 before plunging back down.

So, what does this suggest for changes?

1. Move the decision about what type of cycle to engage in next to the end
of the sweep phase.

2. If memory consumption has grown sufficiently to warrant a full
collection, start it immediately rather than waiting to cross the threshold.

These two adjustments would probably fix the worst of the issues in Mike's
graph.

The behavior of the system also suggests why changing the calculation of
GCthreshold has inconsistent results. Smaller values result in more frequent
incremental collections which tenure data that might otherwise have died in
the nursery. On the other hand, smaller values mean that estimate may not
overshoot twice prevestimate by as much and hence we will do a full
collection sooner.

Mark