Re: Lua workshop

[Tony Finch <dot@...>]

Wim Couwenberg <wim.couwenberg@gmail.com> wrote:

> Back home again after participating in the Lua workshop 2011.  It was
> an inspiring and thoroughly enjoyable event.  So I want to say a big
> thank you to Marc and Vera for making it all happen!

I haven't yet escaped Frick, but I would also like to thank Marc and Vera
for an excellent workshop, and also the speakers (including Wim) for their
wonderful and impressive talks. Here follows my notes of what happened.
I'm going to put a copy on my blog so please excuse the parts where I
explain things about Lua that you already know :-) Also my apologies to
the speakers if I fail to summarise their talks accurately.

I spent the last two days at FiBL, an organic agriculture research
centre, in Frick, a small town between Basel and Zurich. I was there
for the Lua Workshop 2011. There were a bit more than 40 attendees, of
which two were women, with a good spread of ages from about 20-65.
Some themes came up repeatedly: live code upgrades, distributed
systems, Lua internals, LaTeX Beamer slides, and really good demos.

The first day started with Alexander Gladysh, who talked about his efforts
to find a good strategy for implementing declarative embedded domain
specific languages in Lua. Lua was originally designed to be good as a
data description language, so it has quite nice lightweight syntax for
EDSLs. Alexander found that implementing the infrastructure for EDSLs
often got bogged down in spaghetti code and boilerplate. In his talk he
explained how he improved the situation using common data structuring
conventions and traversal algorithms. It was a very high bandwidth talk
with over 60 slides (most containing code) in less than 40 minutes. I need
to go back to read the slides at leisure.

Next was Fabien Fleutot from Sierra Wireless. They make embedded
system monitoring equipment, for checking the performance and status
of things like wind turbines and street lights. The communication
between these in-the-field assets and the back-end monitoring systems
is usually via GPRS, so it's important for them to minimise bandwidth
usage, and they mostly have to rely on the assets to initiate outgoing
connections since their connections are often down or behind NAT.
Their monitoring devices run embedded Lua and make heavy use of
coroutines, with a nice IPC framework and scheduler. Fabien described
how this makes it easier for them to do in-field upgrades and quickly
adapt their products for new customers. They're planning to release
their framework as open source.

After coffee, Gaspard Bucher talked about and showed videos of his
exploits using computers to support live dance and music performance,
including turning his body into a MIDI device so he could use music to
make his muscles contract involuntarily, and attaching motion sensors to
dancers so their movements can be translated into sounds. The latter
required a lot of custom hardware and research into using support vector
machines to distinguish different kinds of movement. He explained how the
pressures of performance seriously damaged the maintainability of his
code. He's now working on Lubyk which is a Lua system with many bundled
libraries, intended to provide a basis for his computer-assisted
performances. It is a distributed development environmnet, using Bonjour
to tie the components together, with a graphical editor to link the parts
and allow you to live-edit the code on remote systems. Very swish, but
still a bit raw.

He was followed by Roberto Ierusalimschy, the principal author of Lua.
He talked about some of the design considerations the Lua team use
when evolving the language, and discussed some of the areas that are
particularly troublesome - table length, varargs, bitwise operations
on floating point numbers.

After lunch Wim Couwenberg of Océ talked about using Lua as a diagnostic
tool for the controller software for their large printers. The controller
software itself is a set of C++ processes running on a Windows PC attached
to the printer. They have an IPC framework with an XML-based IDL, which
includes extensive logging of the activity of the system as a whole. Wim
replaced their log processing tools with a much smaller Lua script which
translates an IDL specification to Lua and runs it to process a log. He
extended the XML IDL with Lua snippets that can verify the interface
requirements have been followed. This improved their ability to debug the
system so much that they are using more IPC to get better visibility into
the system. A nice success using just base Lua without add-ons.

Next was Valerio Schiavoni of Neuchatel University, talking about the
"Splay" system for making distributed application research easier. It runs
on PlanetLab and provides some higher level facilities for distributed
IPC, and makes it easy to push Lua code out to a selection of machines,
run it in a sanbox, collect the data, and produce visualizations of the
results. He did a very smart live demo of a virus propagation simulation
on 200 machines around the world.

After coffee was Peter Cawley, an undergraduate at Oxford University. He
talked about his work finding and exploiting holes in Lua's type system
using cleverly crafted bytecode. By default Lua is compiled without the
assertions that do thorough type checking of values passed by C
embedding/extending code to the Lua API. This means that (for instance)
you can make the low-level table index function try to interpret a string
as a table. The bytecode interpreter is mostly typesafe but there are a
few holes which can be exploited, which Peter described in some detail.
Lua 5.1 has a bytecode verifier which is supposed to make it safe to load
untrusted bytecode, but because of Peter's exploits, Lua 5.2 has no
verifier and instead makes it easy to restrict loaded code to source only,
which is safe provided the library is suitably restricted. Peter has gone
on to write a couple of replacement bytecode verifiers. The first is based
on a type inference algorithm, but this turned out to be too slow to be
practical. The second is based on a simpler analysis of which stack slots
are variables or temporaries or unused, and verifying that they are used
consistently.

The last talk on Thursday was Erik Hougaard talking about the series of
robots called "Crazy Ivan" which he built with his brother-in-law. They
compete in the Danish Technical University RoboCup, which is an annual
obstacle course for autonomous robots. Erik compared the structure of the
competition to a Formula 1 race, in that the competitors have time to set
up their robots to suit the course, and there is a qualification stage
before the final competition. Crazy Ivan's software is customized during
the competition to follow the specific course and win points by performing
tricky tasks. They use Lua to make it faster to adapt the software, in
particular the high level plan of the course and tasks. As well as videos
of previous competition performances, Erik did a live demo of the current
version of Crazy Ivan, whose control computer is a complete PC running
Windows XP with on-board copies of Visual Studio and PIC programming tools
for the robot's peripherals. He ran a remote desktop session from his
laptop to the robot to show the computer vision algorithms it uses to
follow the track and spot golf balls etc. Lots of fun!

On Thursday evening we had an organic wine tasting at FiBL's winery,
with a talk on the diffrences between organic and traditional
viticulture. After that we went for a meal at a local restaurant.

The first speaker the next morning was Gaetan Morice who also works at
Sierra Wireless. He was speaking about the Lua development tools they are
putting together. They are keen to open up their devices for their
customers to program, for which an attractive development environment is
necessary. They have put together a lot of open source components,
starting with Eclipse and adding JnLua, MetaLua, LuaDoc, and a lot of
their own code to produce a pretty swish IDE. They are planning to release
it as open source under the umbrella of the Eclipse Foundation, since they
want to avoid the appearance of trying to lock their cusomers into a
proprietary system. His live demo included lots of nice features like code
completion, highlighting instances of the same variable, and their
debugger.

The second talk of the morning was Roberto again, filling in for
another speaker who was unfortunately ill. This talk was about new
features in Lua 5.2, including more flexible coroutine yields,
emergency garbage collection when memory is low, ephemeron tables to
avoid some memory leaks, lexically scoped global tables, light C
functions, the bitwise operation library, and the goto statement. The
talk covered the rationale for the features and the implementation
challenges for the trickier ones. He's always pleased when an
improvement allows him to delete code.

After coffee, Henning Diedrich talked about d'Arc, which is a performance
oriented extension to the Lua API. He bypassed the standard API in order
to make his JSON serializer faster, since it was a bottleneck in his game
engine. The original PUC Rio implementation of Lua and LuaJIT are similar
enough that Henning could make d'Arc work with both of them. The main
feature of d'Arc is a faster table traversal function. In the standard Lua
API tables are traversed using the lua_next() function which has to
re-check its arguments and re-find its iteration point on every call.
darc_traverse() inverts control so it keeps state and calls down to a fold
function for each table element. There was some discussion between several
people about the wisdom of bypassing the API and whether Lua's tables were
keeping their performance promises.

The next talk was by Ashwin Hirschi, talking about the Reflexis Flow
framework for web applications. This is a declarative EDSL for describing
a workflow between dynamic web pages, intended to be accessible to
non-programmers. Ashwin did an impressive live coding demo putting
together a little web app with a form taking a name, height, and weight,
and calculating the person's BMI. He then added some bells and whistles
including a Google Gauge widget to make the BMI number look more pretty,
and a database search for looking up names from a list of workshop
attendees. It can also draw a graph to visualise the traversals between
pages. He then showed off the traversal graph for a larger application,
which is a meta-recursive web-based editor for Reflexis Flow applications.

After lunch, Patrick Rapin of Olivetti described LuaDura, which they use
for diagnostics and servicing of their printers. Their firmware is written
in C++, and they have a nice reflection system that allows you to make
arbitrary virtual method calls via RPC over the service port. LuaDura
ldownloads a description of all the printer's functionality from the
service port into Lua on a controlling computer. It includes some really
nice readline support so you can tab-complete over the printer firmware
API. It's also multi-threaded, and can control multiple printers at once,
or run in the printer firmware itself. LuaDura is used a lot for
development of printers, and in production for things like flashing the
printer's serial number. Patrick finished his talk by abusing a printer,
making its scanner LEDs shine with random colours, and making the paper
and head motors play the William Tell Overture in two tone polyphony.

Then Francesco Santorini from the University of Basel Hospital talked
about using Lua for processing images from MRI scanners. The development
environment for their Siemens MRI scanner does not encourage
experimentation or ad-hoc image processing pipelines. Their practice
before Francesco developed IceLuva was to transport images by sneakernet
to a system running MATLAB. After plugging Lua into the Siemens software
as a module, they were able to easily translate their MATLAB scripts into
Lua and fit them neatly into the existing system, displaying the output of
the pipeline on the scanner's console. Francesco was sadly unable to bring
the scanner with him to do a live demo, but did have a recording of the
scanner playing Smoke on the Water.

After coffee, Peter Odding spoke about Lua/APR. He started work on
binding the Apache Portable Runtime library to Lua in 2007, at which
time he did not know how to program in C, but approached the project
with the attitude, "how hard can it be?". After learning about manaual
memory management, segmentation faults, and Windows calling
conventions, he released Lua/APR in 2010. One of the problems he had
to overcome is the mismatch between APR's pool-based allocation
strategy and Lua's garbage collector; his solution is to allocate a
pool for each APR object, which is simple if not all that efficient.
Lua/APR includes some shaky prototype multithreading support, which is
still under development. Peter wants to get it to the stage of being
able to implement a performant web server in Lua.

Finally, the last talk was given by the organizer of the workshop,
Marc Balmer. He talked about his project to run Lua inside the NetBSD
kernel. His goals are to use it for rapid prototyping of drivers,
including reverse engineering undocumented hardware. Other
possibilities include scripted autoconfiguration. At the moment he has
the infrastructure for creating Lua states in the kernel, autoloading
Lua modules, and running scripts using this context. His next goal is
to write a working driver for some simple watchdog hardware. One other
significant area of difficulty is multi-processor safety. He has
promised to present a talk at FOSDEM in February on his experiences
writing drivers in Lua.

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
Malin, Hebrides: East or southeast, veering south, 5 to 7, perhaps gale 8
later. Moderate or rough. Occasional rain. Moderate or good, occasionally
poor.