lua_setallocf is the first thing i do after calling lua_open.
Maybe this is the problem.
Why are there lua_open and lua_newstate?
lua_open is a macro defined for compatibility with older Lua
versions. It barely calls luaL_newstate, which is a
convenience function with a default memory allocator.
The core function is really lua_newstate, which is the one you
have to use if you want custom memory allocation.