My original problem was that the the behaviour of luaV_execute() is
different from the way a Lua function returns from OP_RETURN. So I
think maybe this is the only case that is interesting from the point
of view of resolving the original issue.
In my opinion, the difference of the behaviour is the place of results rather than who calls.
When a C function call lua function via luaV_execute(), the place of results always on the top of stack;
If a lua function be called via OP_CALL , the place of results may be in the stack frame if the number of results is fixed. Or it's on the top of stack when the results used as vararg , and it need expands the L->top.
The responsibility of moving results to the right place is luaD_poscall , but it always change the L->top. When the results is in the stack frame, we shouldn't change the L->top, so it returns a signal and let OP_RETURN changes back.