Thank you for your explanations.
Using the registry works well in my tests, and it seems there is no way to break the sandbox from inside.
In case someone is interested in the code, this worked for me:
static int lua_regkey_dtor;
lua_pushlightuserdata(L, &lua_regkey_dtor);
lua_newuserdata(L, 0);
lua_newtable(L);
lua_pushliteral(L, "__gc");
lua_pushlightuserdata(L, my_luaclose_argument);
lua_pushcclosure(L, luaclose_mylib, 1);
lua_rawset(L, -3);
lua_setmetatable(L, -2);
lua_settable(L, LUA_REGISTRYINDEX);
(Combined with reference counting in luaopen_mylib and my luaclose_mylib function.)