Crash in Lua 5.0 on RedHat 9

[Reuben Thomas <rrt@...>]

I still don't have a clue why this should happen when I build dynamically
rather than statically, but I think the problem may be in glibc. By the
way, I have added -fPIC to my build flags.

The reason is this: I have a new stack trace that I'm working on (I
switched on MALLOC_CHECK_=3 to try to find where the problem occurred
rather than where the crash was caused):

free(): invalid pointer 0x804ec80!

Program received signal SIGABRT, Aborted.
0xffffe002 in ?? ()
(gdb) where
#0  0xffffe002 in ?? ()
#1  0x42028b93 in abort () from /lib/tls/libc.so.6
#2  0x420760b9 in free_check () from /lib/tls/libc.so.6
#3  0x42073735 in free () from /lib/tls/libc.so.6
#4  0x4206b831 in fclose@GLIBC_2.0 () from /lib/tls/libc.so.6
#5  0x40038fa2 in aux_close (L=0x804a408) at liolib.c:152
#6  0x40039065 in io_gc (L=0x804a408) at liolib.c:172
#7  0x400204f4 in luaD_precall (L=0x804a408, func=0x804a530) at ldo.c:260
#8  0x400206a2 in luaD_call (L=0x804a408, func=0x804a530, nResults=0) at
ldo.c:311
#9  0x40022561 in do1gcTM (L=0x804a408, udata=0x804e9b8) at lgc.c:410
#10 0x40022618 in luaC_callGCTM (L=0x804a408) at lgc.c:428
#11 0x400283be in callallgcTM (L=0x804a408, ud=0x0) at lstate.c:202
#12 0x4001fbe6 in luaD_rawrunprotected (L=0x804a408, f=0x400283ad
<callallgcTM>, ud=0x0) at ldo.c:88
#13 0x4002843c in lua_close (L=0x804a408) at lstate.c:216
#14 0x08049b54 in main (argc=1, argv=0xbfffe114) at lua.c:435
#15 0x420156a4 in __libc_start_main () from /lib/tls/libc.so.6

As you can see, the problem occurs because fclose is trying to free a FILE
*. But I found that if I put a breakpoint on g_iofile, and run the fopen
in liolib.c:232, then immediately call __free_hook(*pf), to try to free
the pointer, I get exactly the same error as above: invalid pointer. So it
seems that something has already gone wrong within fopen (or there is
something wrong in the malloc system).

Maybe I'm being stupid...

Looking at the source for fopen (yup, I've installed the glibc sources) I
can see that it does malloc a block, but the pointer it returns is
*within* that block...OK, further investigation reveals the pointer that
fclose passes to free (which is the same as that returned by fopen) is a
pointer to the first member of the first member of the struct originally
malloced, i.e. it should be the same address.

Now, I'm unlikely to have found a bug in fopen/fclose! so I think the most
likely thing is that the malloc heap is already corrupted when fopen is
called, so that trying to free the pointer it returns gives an error. I
might now be able to simplify the script that causes the crash:

io.open("/home/rrt/ToDo")

followed by Ctrl+D

causes the crash...(N.B., it happens during lua_close). If the Lua
libraries are static (but glibc is still linked dynamically), it doesn't
happen.

I'm inclined for the moment just to switch to using Lua 5.0 dynamically.
Since I had a similar problem with 4.0 under RH9, I'm inclined to think
something fairly fundamental is wrong (whether in Lua or elsewhere I don't
know). If anyone spots anything I've missed, I'd still be most grateful to
know, of course!

I just tried adding -DLUA_USER_H='"../etc/luser_tests.h"' to my CFLAGS,
but didn't get any extra joy.

-- 
http://www.mupsych.org/~rrt/ | What you don't know controls you