Re: HTTP long polling with Lua

[David Hollander <dhllndr@...>]

> HTTP parser. There are too many ugly corner-cases to be handled

I wanted to go do a more detailed reply on this subject in case anyone
finds it informative. In regard to HTTP parsing corner cases, I
believe the primary concern is doing unbounded reads. For example, in
luasocket I believe there is an option to do read('*l') to buffer CRLF
chunks and return whole lines, similar to io.read(). In the Lua server
Xavante, which uses LuaSocket, we have [1]:

   req.cmdline, err = req.socket:receive ()

Where receive seems to be contained in the COPAS[2] library:

   function receive(client, pattern, part)
       local s, err
       pattern = pattern or "*l"

Which indeed defaults to the '*l' line pattern, without also providing
a parameter for asserting the maximum number of bytes the line should
be bounded too. The typical fuzzing scenario is then to simply
generate infinite length requests of garbage bytes excluding a CRLF
line ending, and tank the server.

A better strategy is to always read a specified number of bytes, use a
state machine, and disconnect if the line ending does not appear in
expected number of bytes, optionally being polite and responding with
413 Request Entity Too Large or 414 Request-URI Too Long.

What's important to consider is that most web frameworks have to
tangle with some parsing anyway, even if they avoid implementing an
HTTP status\header parser. If they want to give the application level
developer granular control over forms, whether a page should accept or
reject uploads, of what type, of what size, and if anything should be
saved to disk or not, the developer will usually have to be smart
enough to write an HTTP request body parser anyway. So the security
correlation could be debatable without actually running a fuzzer or
attempting a pentest.

I'd say there are other big concerns outside of the scope of the
parser, for example spamming disconnects without any information sent,
so the only solution is test coverage, plugging in parsers is no
pancea. That said, lua-http-parser is in fact a binding to the
reliable parser from Node.JS. However, I am using a custom pure Lua
parser in a LuaJIT build not currently on github in order to increase
code reuse for request body parsing and to remove any C dependencies.
This version is not a priority at the moment, and I wouldnt switch to
it without first writing a fuzzer though.

[1] https://github.com/keplerproject/xavante/blob/master/src/xavante/httpd.lua
[2] https://github.com/keplerproject/copas/blob/master/src/copas/copas.lua

On Fri, Jul 22, 2011 at 11:51 PM, David Hollander <dhllndr@gmail.com> wrote:
> It uses the http parser from Node.js which uses the http parser from nginx.
>
> On Thu, Jul 21, 2011 at 6:48 AM, Alexander Gladysh <agladysh@gmail.com> wrote:
>> On Thu, Jul 21, 2011 at 06:47, David Hollander <dhllndr@gmail.com> wrote:
>>>> Looks like it is not possible to do this in WSAPI without some serious
>>>> low-level coding...
>>>
>>> I have a lightweight Lua web server here that uses callbacks instead
>>> of coroutines [1].
>>
>> <...>
>>
>>> Let me know if you are interested in using
>>> this and I'll put up some example applications and documentation this
>>> weekend.
>>
>> This looks interesting, thank you! It would be cool if the ox would be
>> installable from LuaRocks.
>>
>> However, I would prefer to avoid using yet another custom-written HTTP
>> parser. There are too many ugly corner-cases to be handled. So, sadly,
>> no, I'm not interested (yet) in using the ox for production.
>>
>> Alexander.
>>
>>
>