Re: Idea for an event loop library

[Tim Caswell <tim@...>]

Please don't misunderstand me.  Coroutines are very powerful and can do a great many things that callbacks can't.  The big one is preserving the stack!  Stack traces in event-based systems that use only callbacks only go back to the last event source.  If something bad happens early after an event, good luck tracing it back to what started the event.  With coroutines, the stack is paused and when it resumes, you still have the original stack.  This does come with some performance overhead, but I've never seen it be a bottleneck in production.

My only warning is in making the suspension of the coroutine implicit.  You can use coroutines and still be explicit about the points where your code can be suspended.  For example, in Luvit, I added a little sugar called "fiber" that looks like this: https://github.com/luvit/luvit/blob/master/examples/fs-coroutines.lua

fiber.new(function (resume, wait)

  fs.readdir(".", resume)

  local err, files = wait()

end)

In luvit I don't allow suspending the main thread, but you can create new coroutines and suspend those.  All the luvit APIs are callback based, but using the `resume` and `wait` functions you can suspend the coroutine and wait for the callback to be called.

The fiber library is a single 2-line lua function.  Just some simple convention to make things explicit: https://github.com/luvit/luvit/blob/master/lib/luvit/fiber.lua

The one thing faux blocking never did for me is parallel work.  I like my logic to be as parallel as possible to reduce latency for the http request (or whatever operation I'm working on).  Using coroutines to make non-blocking calls appear as sequential blocking calls is great for when you want to do things in sequence. (though I would argue named callbacks using JS hoisting is just as easy)  But for doing things in parallel, it's rather lacking.  Maybe the best is a hybrid of using coroutines when they make sense and callbacks for other stuff.

On Fri, May 11, 2012 at 9:33 AM, Jorge <xxopxe@gmail.com> wrote:

On jue, 2012-05-10 at 20:03 -0500, Tim Caswell wrote:
> My only warning with making the non-blocking operations appear
> blocking is that it will bite novice programmers big time.

Very interesting comments (and others down the thread, too). I'm only
starting to play around with these concepts, and still don't actually
grasp the consequences. Also, my only experience with callback driven
programs was GUI stuff (as a novice programmer, no less) and I found I
very quickly got scared of my own code. Get distracted a split second
and there are those global variables sharing state between callbacks,
ugh.

> Any function call into a third-party library will have the potential
> to suspend your current coroutine.

...

> With implicit suspensions you get into nasty issues similar to thread
> preemption where local state changes out from under you because some
> other event messed with your data.

Ok, I see the problem. The motivation behind Lumen was to see how much i
can do in self-contained, Lua only library, so yes, perhaps I assume a
tighter control of the code than usual. It's just an experiment about
where to place complexity. And, for the only system i've developed using
it (a thing that moves data between files and sockets in various weird
ways) it has so far worked surprisingly well.

Here comes another murky line of reasoning. I was once surprised on how
closures Just Worked, and meant whatever was reasonable in a given
context, even when you as a programmer didn't fully understand the
implications. I wrote an object thing that held the attributes in
closures, without even thinking about it, just because the syntax seemed
natural. And only after that worked and was in production for a while, I
realized it was an object, and didn't have semicolons, etc. So perhaps
i'm just hoping that these data visibility problems will be magically
solved by closures, too :)

Jorge