Re: Non-blocking read from sub-process?

[Daurnimator <quae@...>]

On 6 April 2016 at 12:31, Sean Conner <sean@conman.org> wrote:
> It was thus said that the Great Dan Christian once stated:
>> I want to run a sub-process that may or may-not generate output.  How
>> can I make sure I don't block trying to read it.  I'm happy to use
>> luaposix, but I'm trying not to use a big event framework like lua-ev
>> (because I'm memory limited and also don't want to cross compile it).
>>
>> Something like:
>>
>> f = io.popen(unknown_command, "r")
>>
>> while im_doing_something_else:
>>   data = f:read()
>>   if data:
>>     print ("Subproc says: ", data)
>>
>> kill(sub_proc)
>
>   Okay, I looked into this.  I wrote a simple program:
>
>         #!/usr/bin/env lua
>
>         local clock = require "org.conman.clock" -- [1]
>         local data  = string.rep("a",72) .. "\n"
>
>         for i = 1 , 5 do
>           io.stdout:write(data)
>           io.stdout:flush()
>           clock.sleep(1)
>         end
>
>   Once I had that, I then wrote the following with luaposix:
>
>         local posix = require "posix"
>
>         local cnt = 0
>         local f   = io.popen("./b.lua","r")
>         local fd  = posix.fileno(f)
>         local fds = { [fd] = { events = { IN = true } } }
>
>         while true do
>           cnt = cnt + 1
>
>           local events = posix.poll(fds,0) -- a timeout of 0
>
>           if events > 0 then
>             if fds[fd].revents.IN then
>               local data = f:read("*l")
>               print(data)
>               print(cnt)
>             elseif fds[fd].revents.HUP then
>               f:close()
>               return
>             end
>           end
>         end
>
>
>   And the output:
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 564
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 124517
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 247551
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 370648
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 494827
>
>   This shows that we were able to count up while waiting for more input from
> the running program.  And just because I was curious (because I have my own
> POSIX layer [2]):
>
>         local fsys    = require "org.conman.fsys"
>         local pollset = require "org.conman.pollset"
>
>         local files = pollset()
>         local cnt   = 0
>         local f     = io.popen("./b.lua","r")
>
>         -- -------------------------------------------------------
>         -- select for "r"ead and "h"angup events on the given file
>         -- descriptor
>         -- -------------------------------------------------------
>
>         files:insert(fsys.fileno(f),"rh")
>
>         while true do
>           cnt    = cnt + 1
>           local events = files:events(0) -- a timeout of 0
>
>           for _,event in ipairs(events) do
>             if event.read then
>               local data = f:read("*l")
>               print(data)
>               print(cnt)
>             elseif event.hangup then
>               f:close()
>               return
>             end
>           end
>         end
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 1652
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 379286
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 758243
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 1137190
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 1515319
>
> Hmmm ... seems like my implementation of posix.poll() [3][4] is more efficient
> (higher numbers consistently).
>
>   -spc (Hope this helps some)
>
> [1]     My POSIX layer over the timing functions.
>
>         https://github.com/spc476/lua-conmanorg/blob/master/src/clock.c
>
> [2]     https://github.com/spc476/lua-conmanorg/
>
> [3]     https://github.com/spc476/lua-conmanorg/blob/master/src/pollset.c
>
> [4]     I think this is largly because it will use epoll() under Linux;
>         poll() otherwise, and epoll() has much lower overhead than poll.
>
>         Also, posix.poll() has to scan the given array each time, while my
>         implemetation you only need to add the file descriptors once,
>         regardless of the underlying implementation (select(), poll() or
>         epoll()).
>

If you're looking at anything more than the simplest use of poll() I
urge you to use cqueues:
http://25thandclement.com/~william/projects/cqueues.html

Your program could be rewritten as:

    local fileno = require "posix".fileno -- could be org.conman.fsys
instead but I don't have it installed
    local cqueues = require "cqueues"

    local f = assert(io.popen("./b.lua","r"))
    local pollable = {
        pollfd = fileno(f);
        events = "r";
    }

    local cnt = 0
    while true do
        cnt    = cnt + 1
        if pollable == cqueues.poll(pollable, 0) then -- yield with a
timeout of 0
            local data = f:read("*l") -- this isn't necessarily
correct; if a whole line isn't ready to read it will block
            if data == nil then
                -- f:read returns nil on EOF
                f:close()
                break
            end
            print(data)
            print(cnt)
        end
    end



However, I urge anyone to use the power of coroutines for neater and
maintainable code:

    local fileno = require "posix".fileno -- could be org.conman.fsys
instead but I don't have it installed
    local cqueues = require "cqueues"

    local cq = cqueues.new()

    local cnt = 0
    local notdone = true
    cq:wrap(function()
        local f = assert(io.popen("/home/daurnimator/spc-b.lua","r"))
        local pollable = {
            pollfd = fileno(f);
            events = "r";
        }
        while cqueues.poll(pollable) do -- yield the current thread
until we have data
            local data = f:read("*l") -- this isn't necessarily
correct; if a whole line isn't ready to read it will block
            if data == nil then
                -- f:read returns nil on EOF
                f:close()
                break
            end
            print(data)
            print(cnt)
        end
        notdone = false
    end)
    cq:wrap(function()
        while notdone do
            cnt = cnt + 1
            cqueues.poll() -- yield the current thread
        end
    end)
    assert(cq:loop())


I hope you agree that that is much more readable, and easier to reason about.

For a rather useless comparison, here is the output using my variations:

>From 1st program:
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
932
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
225597
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
448165
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
673735
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
899575


>From 2nd example:
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
6299
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
1426364
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2838288
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
4245709
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
5651244

Though in the 2nd case here we're really just finding out how fast
coroutine.yield() is.