I have a large lua application that works fine with Lua 5.3 and Lua 5.4 that is shipped by my Linux package manager. However I've downloaded the Lua 5.4 sources and compiled myself and when using it I got a crash. I was intrigued because the sources of my system's Lua and the ones I downloaded were equal, just the compilation flags were different. So I decided to test each CFLAGS when compiling to see which one was effecting.
For instance my system compiles Lua 5.4 with the following flags:
gcc -std=gnu99 -O2 -Wall -Wextra -DLUA_COMPAT_5_3 -DLUA_USE_LINUX -DLUA_USE_READLINE -march=native -O2 -pipe -fno-plt -fPIC -flto -g -D_FORTIFY_SOURCE=2
And I was manually compiling with the following flags:
gcc -DLUA_COMPAT_5_3 -DLUA_USE_LINUX -O2 -g
Changing the CFLAGS to the same CFLAGS used by my system my application didn't crash, then I ruled out each flag one by one, and I discovered the following:
Adding -D_FORTIFY_SOURCE=2 makes the crash go away with -O2.
Changing -O2 to -O1 (but not adding -D_FORTIFY_SOURCE=2) also makes the crash go away.
I've tried to use a different compiler, such as clang, and the crash still happens under the same conditions.
I've tried applying patches for all the known Lua 5.4 bugs, but the crash still happens under the same conditions.
And here is the full backtrace with symbols and source location:https://pastebin.com/7ZAL1Edg
In that run -O2 -g flags were used, and I've used sources from the "v5.4.0-patch" branch for lua's github repository.
Unfortunately I don't have a small test case yet.