Re: Luasocket: poll vs select

[Sam Roberts <vieuxtech@...>]

On Tue, Feb 21, 2012 at 7:13 AM, Javier Gallart <jgallartm@gmail.com> wrote:
> I read
> here: http://wiki.voiceworks.pl/display/~pawel/Luasocket+core+dumps+in+socket_waitfd
> that usage of poll instead of select  was preferred. I just tried it and in
> a test environment it woks perfect, I've been able to reach more than 20
> calls/sec without a single failure. However, I'm not really sure of what I'm
> doing and I'd like to know if this change could have any side effect. What
> do you think?

A couple things:

Your problem isn't directly triggered by connect rate, its triggered
by how many concurrently open socket descriptors you have.

If you don't intend to be keeping multiple TCP connections open
simultaneously, then you have a socket leak. Sockets get closed on
garbage collection, of course, but that can take a while. Make sure
you are explicitly closing your TCP sockets!

That alone might fix your problem.

If you do intend to have multiple TCP connections open simultaneously,
then yes, switching to poll() is the right thing to do, it
doesn't have the compile-time set size limitation of select(). I made
poll() the default in our patched luasocket version:

https://github.com/sam-github/luasocket/commit/6af1d4137b26566ba7eae13a5e6faa23c9b9e7c1

Note that an implementation of socket.select() using poll() does not
currently exist, so socket.select() still has a compile-time
limitation on the fd set-size, even if you use -DSOCKET_POLL.

Also, its true, luasocket isn't bounds checking it's calls to
FD_SET(). Its a bug that occurs in two places, and seemingly easy to
fix, except fixing it exposed another bug (error return values
incorrectly converted to strings), and a documentation/interface
question...

Currently, socket.select() ignores socket descriptors that are closed
or invalid (useful, intended, and documented).

However, silently ignoring socket descriptors >= FD_SETSIZE seems like
a landmine in the kitchen, so I could make it return an error (but no
error return is currently documented, though there is one), or I could
make it die with a lua error. I'm tempted to do the latter. Its an
interface change, but it really isn't valid to pass descriptors larger
than FD_SETSIZE to socket.select(), failing hard and fast seems
appropriate.

Anybody have opinions on this?

Thanks,
Sam

p.s. To those suggesting abandoning luasocket for an event-based
approach: I like event based programming, and we've got lots of coding
doing it internally (mostly with twisted), but 20 calls/second isn't
high-performance on anything better than a casio calculator, rewriting
your app into an event framework would be great fun, but not every
little program needs to be event-driven.