[ANN / where Lua is used] Splash: web scraping using Lua

Subject: [ANN / where Lua is used] Splash: web scraping using Lua
From: Mikhail Korobov &lt;kmike84@ ... &gt;
Date: Wed, 4 Mar 2015 20:24:13 +0600

FYI: we're developing Splash (https://github.com/scrapinghub/splash), which is a headless web browser with an HTTP API based on QWebKit; it uses Lua 5.2 as an user-facing scripting engine, i.e. you can write scripts similar to PhantomJS, but in Lua.

It also provides a Lua kernel for IPython / Jupyter (http://ipython.org/), with autocompletion, inline images, syntax highlighting and inspections support. When you start this "Splash kernel" a real browser window appears, which you can control by executing chunks of Lua in a web-based editor. After you're satisfied with a script you can clean it up and parallelize for multiple URLs by submitting it to a HTTP API server.

Most of Splash is written in Python / PyQT; to integrate with Lua https://github.com/scoder/lupa package is used.

We tried to apply to GSoC with Splash this year, but they didn't accept us. However, it is possible that Splash will still participate in GSoC, under Python Software Foundation unbrella.

See http://splash.readthedocs.org/en/latest/scripting-tutorial.html and http://splash.readthedocs.org/en/latest/scripting-ref.html