Re: The Lua interpreter and large scripts

[Mike Pall <mikelu-0911@...>]

Sean Conner wrote:
> -----------------------------------------------
>                 0.00    0.00       1/469488      f_luaopen [136]
>                 0.14    0.00  469487/469488      newlstr [53]
> [62]     0.0    0.14    0.00  469488         luaS_resize [62]
>                 0.00    0.00      12/2101456     luaM_realloc_ [76]
> -----------------------------------------------

Yes, that's an excessive number of calls to luaS_resize. But it
still spends very little time in that function and it only causes
12 actual resizes. So I think it's hitting this check:

  if (G(L)->gcstate == GCSsweepstring)
    return;  /* cannot resize during GC traverse */

What this probably means is that the string table needs to grow,
but can't, because the GC is stuck in the sweepstring phase. It
tries very often (on every string allocation), but never succeeds.
This way the hash chains grow without bounds, which seriously
destroys the performance.

The #1 question is now: why is the GC stuck? Because usually every
string allocation is accompanied with a GC check. And once the
threshold is reached, the GC is driven forward and should quickly
get out of the sweepstring phase.

I think luaX_newstring() in llex.c must be the culprit. You can
try to add a GC check just before the 'return ts' at line 123:
  luaC_checkGC(L);

If this solves the problem, then you've found a bug in Lua.
Congratulations -- it doesn't happen that often. :-)

--Mike