Re: crawling the web

[Norman Ramsey <nr@...>]

 > I am new to Lua, so please bear with me. Does anyone have an example lua
 > script that will scrape a web page e.g extract the video titles from
 > www.youtube.com <http://www.youtbe.com/> .

This is heresy, but on any kind of unix platform I usually have better
luck running curl, either with io.popen or into a temporary file which
I then read.

 > I have tried using http_get(http://www.youtube.com
 > <http://www.youtube.com/> ) 
 > I have had a look in the http.lua module and this function doesn't
 > exist.

It would be a great function to have, but the Lua philosophy is that
you get parts, out of which you build what you need.  One day there
will be a successful effort to build a rich programming environment
for Lua, and on that happy day the APIs will be so wide that nobody
will be able to learn them.


Norman

P.S. If I sound cranky, it's because I've spent 2 years working on
APIs for email management and spam filtering, and they are still too
complicated.  I will give a talk at the Lua Workshop in July.