Re: C arrays as Lua tables

[Dimitris Papavasiliou <dpapavas@...>]

> - In LuaCocoa (a Obj-C/Lua bridge), there are some APIs that deal with
> C-arrays. When crossing from Cocoa to Lua, I push lightuserdata
> (pointers). The idea is that when going back from Lua to Cocoa, I can
> send that pointer through, but in Lua, I cannot access that data at
> the moment. Nor can I create new arrays in Lua to be passed to
> C/Obj-C. I could invent my own userdata type, but if I want to
> interoperate with other Lua 3rd party libraries, I'm kind of stuck.
> I've been putting off implementing my own hoping I can find a solution
> that allows more interoperability with other libraries.
>
> For example, when dealing with OpenGL or OpenAL Lua bindings, those
> APIs also want buffers. They have their own userdata systems which
> would be incompatible with any I invent for LuaCocoa (or any other
> project.) FYI, I have my own OpenAL Lua binding which is also waiting
> for me to solve this interoperability problem.

As I mentioned that was one of the reasons why I made this into a
separate package.  I basically needed to pass large arrays of numbers
to OpenGL and OpenAL but I didn't like them to be opaque objects as
far as Lua is concerned.  Passing them as normal Lua tables is not an
option really for various reasons but mainly due to memory consumption
and increased load time.

As far as interoperability with other APIs is concerned, obviously the
best way to achieve it is to try to create a common C array
representation for Lua that most APIs can adopt.  I think though that
you could interface larray or some other similar C array
representation with APIs that pass arrays as light userdata around
like you do.  If struct array (see core.c) is reorganized slightly so
that the union holding the values is placed first then the pointer to
the struct array held in the userdata will be a pointer to the C array
itself (plus some other stuff at the end which is of no importance) so
it can be used in place of the light userdata.

> Anyway, these are my initial thoughts about what I think might need to
> be in such a library.
>
> - A new userdata type for C-arrays.
>
> - I think a matrix type should be a separate library

What is the difference between a matrix and an array?  A matrix can be
encoded in C or Lua as an array of either one ore two dimensions.  In
any case it's the same thing really.  Arrays of any dimensionality can
be represented with the same userdata in larray and they can be
accessed in the manner one would access them if they were normal C
arrays or Lua tables.  A matrix in this context is just an array of
arrays (the row vectors).

In addition having the same type for arrays of any dimensionality
allows for some tricks which might come in handy.  For example it is
possible to recast a 100 x 100 array representing an image into a 10 x
10 x 10 x 10 array (with virtually no cost).  This can be seen as a
10x10 array of tiles each consisting of 10x10 pixels.  These could be
then in turn be used just as the whole image would be used so you
could say something like blur (a) to blur the whole image or b =
array.cast (10, 10, 10, 10, a); blur(b[1][1]); to blur just the upper
left corner.  Similar things apply to sound arrays or
vectors/matrices.

> - I think indicies should should start counting at 1 for consistency
> in Lua, though maybe a runtime property can be set to change this if
> it is a real sticking point for some people

Indices do indeed start at 1 in the attached implementation but that
can be easily made configurable if required.

> - I don't know if there is some kind of API compatibility interface
> that can be created to work with LuaFFI. (But I don't know how
> userdata checks would ever work.)

I suppose that wouldn't be too hard if it proved necessary but then
again I have no idea how LuaFFI encodes arrays.

> - Memory management I think is the most challenging part. There are
> cases where you might want the memory to be managed through normal Lua
> memory management (like a regular Lua table), but there are cases
> where you are creating the memory to feed to a C-API (such as an
> OpenGL texture or OpenAL buffer). These may need to outlive the Lua
> state that created it. In addition to specifying the management mode,
> it seems you might also want to specify the memory allocator, e.g.
> malloc/free, new/delete, Lua allocator, some custom allocator.

I think that the memory should in general be handled by the userdata
itself otherwise you're in for a heap of leaks and segmentation
faults.  If the data is to be fed to some other API it should be
copied by it like the OpenGL and OpenAL do or alternatively an
explicit reference to the userdata value should be kept to keep it
from being collected until the data is no longer needed.  An exception
is when you want to expose some existing data (either part or all of
it) as an array in Lua so collecting this array should not free the
data.  This can all be handled in larray as you can specify whether
the array data should be freed upon userdata collection or not.

Specifying the allocator should be just a matter of replacing
malloc/free in the source with some macro which can be redefined to
any desired alloc/free pair.

Dimitris