lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


A cookie jar to store cookies is for something else: it does not create "keepalive" HTTP sessions, but allows restarting sessions that have been already closed by proving again the stored cookies.

No, there's NO need at all of ANY cookie jar storage in HTTP to use "keepalive" sessions.

So this just looks like a limit of the current "http.*" package you use which terminates all sessions immediately after each request instead of maintaining it (note: maintaing a session is not warranties: the server may close it at any time if your session is idle for too long, or for various reasons, but for HTTP sessions you want to use for example to performed streamed requests in a queue, reusing the session will always be faster than creating one new outgoing session for each request in your queue (including the cookies exchange which also add to the data volume transmitted).

Without "keepalive", if you want to perform a lot of small queries in a long series of requests, or want to do streaming, your client would rapidly exhaust its number of available TCP ports (because each new HTTP session allocates a new port number, and even if it is closed and the server has acknowledge that closure, that port number cannot be reused immediately before a delay which is generally about 30 seconds (this varies if you have proxies, or depending on your ISP or router, or depending on security thresholds: for security and privacy of TCP, this delay should never be reduced too much: there are security routers that force outgoing TCP port numbers to remain in FIN_WAIT state for more than 3 minutes, and if your Internet connection is shared by multiple users or other services, you'll fall to a case where you no longer have any usable local TCP ports to allow more connection, and your repeated HTTP queries will rapidly then fail to connect after a few thousands queries performed rapidly).

The "keepalive" feature was made a *standard* part of HTTP/1.1 (instead of being optional and not working very well in the now very old HTTP/1.0 protocol) exactly to preserve network resources, and get much better performance and lower bandwidth usage. Without it, most modern websites would be extremely slow.

Using "curl" to create a cookie jar does not mean that you are using keepalive, but at least it allows all requests you send using the same cookie jar file to reopen new sessions using the same resident cookies that were exchanged in the previous requests (so this "simulates" sessions, but still you'll see many outgoing TCP sockets that are "half-terminated" in FIN_WAIT state and that still block the unique port numbers that were assigned to that socket).

So I suggest instead using a virtual "local HTTP proxy" package that will maintain a session opened on an unterminated HTTP session, overwhich you'll queue one or multiple http requests. Such thing is used in (de)muxing subprotocols (of multimedia streaming protocols, or in VPN protocol managers), but it is even part of classic web browsers that queue the many queries that are performed in the background when visiting a website (generally a browser can create up to 4 HTTP sessions working in parallel, each one having its own queue of queries). All of these use the keepalive feature of HTTP/1.1.


Le jeu. 8 nov. 2018 à 08:49, Dirk Laurie <dirk.laurie@gmail.com> a écrit :
Op Do., 8 Nov. 2018 om 02:47 het Srinivas Murthy
<the.srinivas.murthy@gmail.com> geskryf:
>
> This is an important issue for me. Can anyone please add to this? There's got to be a way to avoid conn setup/teardown each time we issue a http.request()
>
> Thanks
>
> On Wed, Nov 7, 2018 at 12:44 PM Coda Highland <chighland@gmail.com> wrote:
>>
>> On Wed, Nov 7, 2018 at 1:56 PM Dirk Laurie <dirk.laurie@gmail.com> wrote:
>> >
>> > You need to provide a file for stroing cookies.
>> >
>> > I can do it from the module 'curl' installed by 'luarocks install lua-curl'.
>> >
>> >   local session = curl.easy()
>> >   local cookies
>> >   if curl.OPT_COOKIEFILE and curl.OPT_COOKIEJAR then
>> >     cookies = os.tmpname()
>> >     session:setopt(curl.OPT_COOKIEFILE,cookies)
>> >     session:setopt(curl.OPT_COOKIEJAR,cookies)
>> >     logins[session] = cookies
>> >   else
>> >     message("Your 'curl' does not support cookies. You will be anonymous.")
>> >   end
>> >
>> > The module 'http' should offer a similar service.
>>
>> That's curious. A cookie jar is required in order to use HTTP
>> keepalive? Is that documented? It seems like a dubious dependency.

All I can attest to is that curl with a cookie jar works. It might
well be overkill of you need only keepalive, but since I use curl only
for sites that require me to login, I would not know.