Re: The Lua interpreter and large scripts

[Roberto Ierusalimschy <roberto@...>]

> > > The #1 question is now: why is the GC stuck? Because usually every
> > > string allocation is accompanied with a GC check. And once the
> > > threshold is reached, the GC is driven forward and should quickly
> > > get out of the sweepstring phase.
> > > 
> > > I think luaX_newstring() in llex.c must be the culprit. You can
> > > try to add a GC check just before the 'return ts' at line 123:
> > >   luaC_checkGC(L);
> > > 
> > > If this solves the problem, then you've found a bug in Lua.
> > > Congratulations -- it doesn't happen that often. :-)
> > 
> >   I tried that, and nope.  Still takes six minutes.  
> 
> Nevertheless it would be useful to check Mike's theory. A simple (and
> somewhat dirty ;) way to do it would be to add the following line
> at the beginning of luaS_resize (just after variable definitions):
> 
>   fprintf(stderr, "gcs: %d strgc: %d size: %d nuse: %d bytes: %d tsh: %d\n",
>         G(L)->gcstate, G(L)->sweepstrgc, G(L)->strt.size, G(L)->strt.nuse,
>         G(L)->totalbytes, G(L)->GCthreshold);
> 
> (That will generate 469488 lines of output. But any chunk after the
> first thousand lines could give us some clue of what is going one. You
> can abort the run after the first few thousand lines.)

Adding those lines confirmed Mike's explanation. The GC is stopped in
the sweepstring state, and so nuse keeps growing without resizing the
string table (strt.size). However, adding the correction he suggested
("luaC_checkGC(ls->L);" on function luaX_newstring in llex.c) solved
the problem here.

-- Roberto