Non-blocking HTTP request with socket.http and copas

[Paul K <paul@...>]

Hi All,

I've been using copas for non-blocking socket processing for quite
some time, but until recently couldn't make it work with requests that
involve socket.connect. It turned out that (1) copas API is not fully
compatible with socket API that it wraps, and (2) socket.connect and
copas.connect don't play well with non-blocking connect requests [1].
The solution from PiL [2] works fine for non-blocking send/receive,
but blocks on connect. I could extend it with settimeout logic, but
didn't want to do all http processing myself. I also tried the
solution suggested by Diego in this thread [3], but it didn't quite
work for me and I wanted to use copas if possible.

Here is the request function I came up with. It implements a
workaround to address socket.connect issue (as described here: [4]):

function asyncrequest(url)
  local socket = require('socket')
  local copas = require('copas')
  local http = require('socket.http')
  local ltn = require('ltn12')

  local timeout = http.TIMEOUT
  http.TIMEOUT = 0.01
  local body, status, headers, response
  local t = {}
  copas.addthread(function()
      local request = type(url) == 'table' and url or {url = url}
      request.sink = request.sink or ltn.sink.table(t)
      request.create = request.create or function()
        local wrapped = copas.wrap(socket.tcp())
        wrapped.connect = function (self, ...)
          local status, err = copas.connect(self.socket, ...)
          if status == nil
          and err == "Operation already in progress" then
            socket.select({}, {self.socket})
            return true
          else
            return status, err
          end
        end
        return wrapped
      end
      body, status, headers, response = http.request(request)
      http.TIMEOUT = timeout
    end)
  return function()
    if not status then copas.step(0); return false end
    return body and table.concat(t,"") or nil, status, headers, response
  end
end

local start = os.clock()
local done = asyncrequest("http://studio.zerobrane.com";)
print("started request", os.clock()-start)
while not done() do end
local body, status, headers, response = done()
print("done", os.clock()-start, status, #body)

You also need a patched version of copas [5] that extends socket API
to make wrapped copas sockets to work with socket.http "open" method.

It works well for me on Windows, but still blocks on requests for
non-existing addresses.b

The current logic also doesn't work when http.TIMEOUT is set to 0
instead of a small non-zero value. The result is still sufficiently
good for me, but I'd be interested to know how to make it work for 0
timeout as well.

The workaround using "create" method won't be needed when #99 is fixed
in luasocket. Let me know if you see any flaws with the implemented
logic.

Paul.

[1] https://github.com/diegonehab/luasocket/issues/99
[2] http://www.lua.org/pil/9.4.html
[3] http://lua.2524044.n2.nabble.com/LuaSocket-COPAS-CoSocket-td7627014.html
[4] http://lua.2524044.n2.nabble.com/LuaSocket-asynchronous-connect-on-Windows-not-behaving-properly-td3981262.html
[5] https://github.com/keplerproject/copas/pull/18/files