Re: Lua 5.4.0-rc5 segfault in low memory conditions

[Roberto Ierusalimschy <roberto@...>]

> Andrew Gierth <andrew@tao11.riddles.org.uk>:
> > I think there are more barriers missing in lundump.c, specifically when
> > string values are loaded into the constants vector (and debug vector) of
> > a prototype; this needs a barrier (since the prototype might have been
> > marked) but I see no sign of one.
> 
> Though I'm a bit uneasy sprinkling barriers in so many places where
> there used to be none, looks like the attached (updated) patch
> mitigates the problem, even for cases of multiple failures on MIPS.

Indeed, there seems to be several barriers missing in lundump.c, besides
that one in lparser.c. I am still not sure why this happened, but I
suspect this comes from before emergency collections. Without emergency
collections, lundump.c never runs the GC, and therefore it didn't need
barriers.  When we added emergency collections, there was great concern
about everything being properly anchored all the time (and lots of tests
around that issue), but we did not focus on the problem of barriers.

The following setup seems to catch several (all?) of these errors:

- Add calls to lua_checkmemory around the emergency collection:

    static void *tryagain (lua_State *L, void *block,
                           size_t osize, size_t nsize) {
      global_State *g = G(L);
      if (ttisnil(&g->nilvalue)) {  /* is state fully build? */
    lua_checkmemory(L);            /* <<<<<<<< */
        luaC_fullgc(L, 1);  /* try to free some memory... */
    lua_checkmemory(L);            /* <<<<<<<< */
        return (*g->frealloc)(g->ud, block, osize, nsize);  /* try again */
      }
      else return NULL;  /* cannot free any memory without a full state */
    }

- Compile Lua with -DHARDMEMTESTS

- Use a script like this one to load binary chunks:

     local f = assert(loadfile(arg[1]))
     local b = string.dump(f)
     f = assert(load(b))
     f()

Until now, I found a barrier missing when loading prototypes and when
loading strings, as already reported in this discussion. (I also
found a small bug in lua_checkmemory itself :-)

Again, many thanks!

-- Roberto