Re: Blocking calls and module composability

[Sean Conner <sean@...>]

It was thus said that the Great Archie Cobbs once stated:
> Hi Sean, thanks for your reply.
> 
> On Sat, Jan 25, 2020 at 6:15 PM Sean Conner <sean@conman.org> wrote:
> > > The most profound restriction is the limitation to a single C call
> > > stack (you can't assume pthreads), and so Lua must longjmp() every
> > > time there is a coroutine context switch.
> >
> >   The Lua call stack and C call stack are two separate things.
> 
> Sorry if I wasn't being clear. Still learning all this stuff.
> 
> I was referring to this section 29.2 of "Programming in Lua" (4th ed):
> 
>     It is easy for the interpreter to have multiple soft stacks,
>     but the runtime of ISO C has only one internal stack.
>     Therefore, coroutines in Lua cannot suspend the execution
>     of a C function: if there is a C function in the call path from
>     a resume to its respective yield, Lua cannot save the state
>     of that C function to restore it in the next resume....
> 
>     Lua 5.2 and later versions ameliorated that difficulty with
>     continuations. Lua 5.2 implements yields using long jumps,
>     in the same way that it implements errors....

  Good lord!  It even mentions it in the Lua 5.2 manual ... how did I miss
that?  Sigh.

> So I'd first ask: What is the single most important problem that is
> inherent and inescapable with "plain" Lua that we'd like to solve with
> additional capabilities? And what is the minimum additional capability
> that could be required in order to solve it?
> 
> In other words, what's the lowest hanging fruit here?
> 
> To me the most important such problem is the non-composability of modules.

  I would agree with that.  I know that when I wrote my own modules [1], I
try to focus them to a single concept instead of writing a large
encompassing module.  I've also made sure they work well together.

> > > In my view, Problem #2 is the most serious problem. It eliminates a
> > > huge class of applications, namely, any application supporting
> > > simultaneous blocking operations. For example, a web server!
> >
> >   It does not.  At work, I have a program written in Lua that is handling
> > over 200,000,000 calls per day [a].
> 
> But surely you're using luv or cqueues or some equivalent, right?
> 
> Or do your Lua VM's still block completely when you read from the network?

  Something equivalent to luv or cqueues (I wrote my own).  And no, the Lua
VMs I run don't block when I read from the network.  I took what you said as
"you can't write a network server in Lua."  

> > > OK so far so good...  BUT - there's still a larger problem with "luv"
> > > or any other solution to Problem #2 I've run across so far: none of
> > > them are COMPOSABLE with other, unrelated Lua modules.
> >
> >   True, but you have that issue in any language.
> 
> Not really, at least as it relates to Problem #2.
> 
> Take POSIX C for example. I can write a C shared library with a
> function F() that invokes a blocking system call, and my library can
> be linked into some random program, and F() can be invoked without
> freezing the entire program. POSIX C provides a standard way to handle
> this (pthreads).

  Yes, and my colleagues at work had a hell of a time integreating a DNS
library into their projects, even *in* the presense of pthreads, because the
library they selected wanted to do DNS queries its way, existing program be
damned!

> An even more interesting example in this debate is Node.js. They solve
> this problem simply by making it impossible to invoke a blocking
> system call directly!
> 
> But look how composable Node.js is, and how its adoption exploded. I'd
> argue composability was an essential enabler of that growth.

  But with callback hell and promises (a form of callback hell).  I'd call
Javascript popular because it comes with every browswer out there, not
because it's any good.

  And Node.js has other issues (leftpad anyone?).

> >   What functions would those be?  Because there's this other thread trying
> > to define a minimal set of "batteries" for Lua ...
> 
> Here are the POSIX functions that pth provides wrappers for:
> 
> pth_nanosleep, pth_usleep, pth_sleep, pth_waitpid, pth_system, pth_sigmask,
> pth_sigwait, pth_accept, pth_connect, pth_select, pth_pselect,
> pth_poll, pth_read,
> pth_readv, pth_write, pth_writev, pth_pread, pth_pwrite, pth_recv, pth_recvfrom,
> pth_send, pth_sendto.

  Oh, so I have to rewrite code to use the new names (or a preprocessor
hack).

  -spc

[1]	https://github.com/spc476/lua-conmanorg