Second draft of new Event API for Luasocket

[Mike Pall <mikelu-0406@...>]

Hi,

the second draft is available at:

http://lua-users.org/files/wiki_insecure/users/MikePall/event.html

Based on your input the API has changed quite a bit. A big thank you goes
to everyone who has provided me with the required information for
'Windowizing' the API!


Ok, here is the long-winded explanation for the changes (and there are
still some open issues):

It seems the only way to properly do event-based I/O on Windows is to
convert Luasocket to use overlapped I/O exclusively. Alas, the POSIX
emulation Winsock gives us is just an illusion. And a bad one, too.

The changes in the Windows specific backend (wsocket.c) amount to replacing
recv() + select() with WSARecv + WSACreateEvent + WaitForSingleObjectEx +
WSACloseEvent. We may be able to recycle the event handle though and call
WSACloseEvent when the socket is deleted. Dito for recvfrom(), send() and
sendto(). We need to use WSAEventSelect for accept() and connect()
and get better error handling for connect() with WSAEnumNetworkEvents.

I assume this is at least as efficient as what select() does behind the
scenes. But this *only* takes care of the standard socket timeouts set
with settimeout(t) and t > 0.

Sockets to be used with an event multiplexer always need to have a zero
timeout. And there may be many of them. The approach using Windows handles
is unsuitable in this case, because the WFMO-style functions handle only
up to 64 (or 63) handles. The only way to allow for more sockets is to add
a completion handler to the overlapped I/O operation and somehow trigger
a virtual event.

So I have devised the following approach:

- The event API has been extended for virtual events. This is basically
  a simple condition and can be in the untriggered or the triggered state.
  Creating and using virtual events is cheap. They can be safely triggered
  from inside an APC (such as a completion handler from an overlapped I/O
  operation).

- The C API to trigger a virtual event needs only the event id. Neither the
  Lua state nor the event container is needed. This allows for passing the
  virtual event id as the hEvent member of the [WSA]OVERLAPPED structure
  when a completion handler is specified (this is a recommended re-use
  of this member according to MSDN).

- A socket can be put into 'event mode' by sock:seteventmode(ev). The
  event container to be used must be registered with the socket (see
  below for some problems with that and an alternative).
  The socket needs to be put into non-blocking mode, too (which is
  automatic with some calls like WSAEventSelect).

- All socket calls that allow for overlapped I/O need to be changed
  (pseudo code follows):
  
  if (event mode is set) {
    add the completion_callback to the WSAOVERLAPPED structure
    invoke overlapped socket operation (e.g. WSARecv())
    if (operation has completed immediately) {
      return the result
    }
    create a new virtual event using the event container
    set the event id in the WSAOVERLAPPED structure
    return "event" error plus the event id
  }

  completion_callback(... WSAOVERLAPPED ...)
  {
    trigger the event using the event id from the WSAOVERLAPPED structure
  }

- Neither connect() nor accept() allow for overlapped I/O. This means they
  need to use the Windows handle model in event mode, too:

  if (event mode is set) {
    use WSACreateEvent to create a Windows handle (if not done, yet)
    use WSAEventSelect, pass the handle in and set FD_CONNECT or ACCEPT
        plus FD_CLOSE
    invoke socket operation (e.g. connect())
    if (operation has completed immediately) {
      clear the event bits with WSAEventSelect (may be omitted for accept?)
      return the result
    }
    create a new event for a Windows handle using the event container
    return "event" error plus the event id
  }

- For orthogonality here is the way the POSIX socket backend needs to
  be changed:

  if (event mode is set) {
    invoke socket operation (e.g. recv() or connect())
    if (operation completed immediately) {
      return the result
    }
    create a new event for POSIX file readability or writability using the
      event container
    return "event" error plus the event id
  }

Ok, so far so good. Now the question to the Windows experts. Does this
work? Are there any pitfalls?

About IOCP: Unless I'm mistaken this involves creating additional threads
and is only useful if it would be too tedious to put the logic into an APC.
A completion handler has better portability and is more efficient
(my guess). And the completion handler needs about one line of C code ...

About AcceptEx/ConnectEx: I'm not sure if the above approach with
completion handlers and a zero receive/send size works with them. And they
are not available under 95/98/Me, too. Having more than 64 listening
sockets is unlikely. But having more than 64 outstanding connecting
sockets is not uncommon (proxies, outgoing SMTP, ...). *Sigh* ...

And here is the stuff that will probably make Diego unhappy:

- The above way using a special event mode is the only way I could think
  of that works with overlapped I/O. Generating the event id is OS specific
  and must be done from inside the socket call (the event id must be
  passed to the completion handler, too). This simply cannot be done with
  settimeout(0) and a Lua wrapper around the call.
  For this to work the event container must be passed in. This can be done
  on each socket call or by setting a special 'event mode' and passing
  the container object once.

- The latter requires storing a Lua object for each socket. Since USERDATA
  objects cannot have associated Lua objects (uh oh, there's the problem
  again) we need to use a table. Using one table per socket is a bad idea.
  A common table that maps socket objects to event container objects needs
  to be created. This can be passed as an upvalue for all socket calls.
  For efficiency the private socket structure needs a flag to indicate
  that event mode is set and that the mapping is to be managed.

  I've used it in the above examples and the docs, but honestly, I don't
  like it much ...

- The alternative would be to add an optional argument to each socket
  call that supplies the event container. This would not work for send()
  though, since it takes multiple arguments (well, we could change
  the API ...).
  Or we could insert a 'magic' second argument (after the socket) and
  have each socket call check if this is USERDATA and assume this is
  an event container. In this case all other arguments are shifted by one.
  Else the standard argument positions apply.

  This is somewhat hacky, but pretty simple to implement. And avoids
  storing and fetching the event container object from a table.

- Anyone have a better idea?

- The return from an socket call needs to include an indication that
  an event has been created plus the event id. I think the only efficient
  way is to add the event id as an additional return value after all
  other return values and set the error to "event". Using the fact that
  the event id is a LIGHTUSERDATA object could be used to return it
  instead of the error string. The surrounding Lua code needs to check
  for type(err) == "lightuserdata" then, which is probably slower than
  checking for err == "event".
  
  Oh well ... I'm not too happy with this one either.

Bye,
     Mike