lua_setallocf is the first thing i do after calling lua_open. Maybe this is the problem.
Why are there lua_open and lua_newstate?
lua_open is a macro defined for compatibility with older Lua versions. It barely calls luaL_newstate, which is a convenience function with a default memory allocator.
The core function is really lua_newstate, which is the one you have to use if you want custom memory allocation.