A question about event driven paradigm and helper/worker thread

[gary ng <garyng2000@...>]

Hi,

I have been working with COPAS recently and it in
general works pretty well if the major blocking
operation(socket/pipe, or regular file read/write) can
run in non-blocking mode and monitored by select() or
libevent.

However, there is one common source of pontential
blocking operation, SQL request, which I don't know
how to handle, even with helper thread and/or
lua-lanes.

The scenario is like this(say for a typical http
request) :

read the request
do some processing
response.

Now if the processing is fast and the response only
involves file i/o, I can bypass the standard io
library and use non-blocking read/write with select()
to ensure it would not block other coroutines(say for
example the underlying may be a NFS mounted file
system).

But if the "do some proccessing" is a SQL request, I
don't know how to handle it. The obvious answer is,
use HTT or lua-lanes to encapsulate it and check the
status of it on each loop and continue when ready.

The problem though is, with such a design and I have
say 10K connections, I need 10K helper threads which
totally negate the advantage of the orginal event
driven design, where each coroutine takes only about
20K(for a hello world type response) vs 1M for an OS
thread.

A modification of it would be to have a pool of helper
threads and find the free one and use it, then spawn
new ones if all are occupied.

This would then mean I am redoing what apache does and
it that case, why not just use apache/lighty +
fastcgi(in dynamic mode) ?

It seems that a solution like Erlang where the threads
are very lightweight(could be even lighter than lua)
yet is preemptively multitasked inside the VM(rather
than using the OS like HTT where we are back to the
thread needs quite some stack space) can really make
use of event driven paradigm yet scale well ?



       
____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow