Why I can't resume a coroutine created in a previous call to a C function?

[Thiago Padilha <tpadilha84@...>]

Hi

I've recently created a msgpack-rpc client library in lua, and it includes a C
module that uses libuv event loop to read/write data from/to the server without
blocking the main thread.

The library is very simple and organized in three core layers:

- The event loop layer, which is implemented in C. This layer exposes a function
  for registering a callback that is invoked whenever the server sends data. It
  also exposes a non-blocking function("send") for sending data to the server.

- The msgpack layer, which wraps the event loop layer and has similar API,
  but operates on "messages" instead of strings: The registered callback is
  invoked whenever a msgpack payload is deserialized, and the "send" function
  receives arbitrary messages(which can be any lua object) that are serialized
  and passed to the event loop layer.

- The msgpack-rpc layer, which wraps the msgpack layer and has a similar API,
  but operates on requests/responses instead of messages: The registered
  callback is invoked when the server sends a request. The "send" function
  receives arguments to a client->server request, including a callback that is
  invoked when the response is available. (To understand some msgpack-rpc terms,
  a skim through the spec is recommended:
https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md)

I've also created an extra layer that wraps the non-blocking msgpack-rpc layer
with a blocking API(let's call it "synchronous msgpack-rpc layer"). Here's how
it works:

- Each request received from the server is handled in a separate coroutine
- The "send" method yields the current coroutine when the request is sent, and
  the response callback resumes it, creating the illusion of a blocking call for
  client code running under the coroutine.

Here's a full example of how the call graph may flow:

  1. The server sends a request.

  2. The data callback, which is provided by the msgpack layer, is
invoked by libuv
     event loop(from C)

  3. The msgpack callback, which is provided by the msgpack-rpc layer, is
     invoked by the msgpack layer.

  4. The msgpack-rpc layer parses the message as a request and invokes
a callback
     provided by the synchronous msgpack-rpc layer.

  5. The synchronous msgpack-rpc layer creates a coroutine to handle the request
     and invokes a client handler function.

  6. The handler function sends a request to the server using the synchronous
     API, which will invoke 'coroutine.yield' after the request is dispatched.
     A callback to be invoked when the response is available is also passed to
     the layer below.

  7. The call returns back to the C event loop.

  8. The server sends a response, which will cause steps 2 and 3 to repeat.

  9. The msgpack-rpc layer parses the message as a response and invokes the
     response callback passed through the client handler in step 6

 10. The response callback resumes the pending coroutine, which was created in a
     previous event loop iteration(So it switched back to C code after the
     coroutine was created).

The problem with the above approach is that I've bumped into the famous
"attempt to yield across metamethod/C-call boundary" error. The error doesn't
happen when I apply Mike Pall's Coco patch: http://coco.luajit.org/ or use
luajit.

So, I have the following questions:

- Why Lua has this limitation? I can't understand why a coroutine can't be
  created/yieled by call coming from C, and resumed after the C call returns.
- Is there a way I can reorganize my code work around this issue without relying
  on a patched lua VM or on luajit?(I want to publish the code to luarocks)

I will also take this opportunity to congratulate everyone involved in Lua
development, it's a great piece of programming art!

Best regards

Thiago