I'm a bit late, but just wanted to mention that when LTCG and the default 1 MiB
stack size is used with some MSVC versions (details below) to compile a 64-bit
Lua, the cstack.lua test fails at the simple recursion. Looks like the
generated code from this option uses up more stack space and the default
LUAI_MAXCSTACK value of 2000 is too high for it. I'm guessing LTCG is not used
often enough that it should be taken into consideration for the default value,
The linker has a /STACK parameter to set that stack size. /STACK:0x200000 should set it to 2MB. This only works for .EXE files.
I don't currently have access to a Windows machine, but I remember doing this in the past.
BTW: I've built Lua with link time optimization on Linux many times, and this problem does not occur, even if you set LUAI_MAXCSTACK to 10000.