Re: typed arrays

["M. Edward (Ed) Borasky" <znmeb@...>]

On Mon, Oct 1, 2012 at 6:55 PM, Peter Loveday <peter@loveday.org> wrote:
>> Since there hasn't been any response from anybody else who actually builds
>> extensions
>> relying on numerical arrays in Lua on this list, it looks like there is
>> little interest or
>> community for this in Lua. Thanks again to everybody who responded.
>
> I imagine there are plenty of users who may well use or write extensions or
> interfaces relying on numerical arrays.
>
> I can see some benefit in a standard, certainly, and anyone who’s into
> serious computational stuff these days it would have to be FFI based for
> speed, for a start. Certainly there could be wrappers on top of that to
> define a standard, that’s not too much of a problem.
>
> As a Visual Effects / Image Processing application, we certainly do, and I
> guess I can only speak from that field, but I think, for us, I’m just not
> sure how useful it would be, unless I’m missing the point somewhat.
>
> Dealing with multiple image processing APIs, SDKs, Formats, Applications
> etc, that we need to support, I can’t rally see too much benefit... is the
> data 8 bit/channel, 16 bit? 16 float? 32 float? 10 bit linear? 10 bit log?
> Does it have Alpha, or is it Alpha only?  Is the alpha packed in or as a
> separate buffer?  Are there additional channels such as Z, Normals, etc, and
> what format are they in?  Is the data BGRA, RGBA, or some other format?  Is
> the image buffer a contiguous block or an array of pointer per scanline?  Is
> it big or little endian data, and if endian conversion is required should it
> be split on channels, pixels, or 32 bit words?  Is the origin top left or
> bottom left?
>
> These are all issues we have to deal with when handing various different
> APIs and formats of data (yes it could all be normalised, and in many cases
> is, but there are unacceptable speed/accuracy tradeoffs in other cases); and
> so I suppose for us the big issues is all about data representation format
> standardisation, not array format standardisation.  And that’s not really
> going to happen any time soon.
>
> An array format is great, I have no issue with that, but it’s not like we
> would be able to simply pass it from one component/extension to another and
> expect it to work without knowing how to interpret the data anyway.
>
> Again this is just for our particular use case / industry.
>
> Love, Light and Peace,
> - Peter

People whose application is *dominated* by number crunching / array
processing will most likely be using (in the open source world) R,
Octave, NumPy, SciPy, Sage or something similar rather than Lua,
JavaScript or Perl. I do nearly everything in R myself because the
libraries are better than anything else in the open source world. You
have to go to Matlab or Stata to get anything that good, and I'd be
surprised if they're *better*, with the possible exception of signal
and image processing.

Moreover, it's very easy to set an R process up as a server and send
it data and code. If you have a lot of existing Lua code, you can
buffer data in and out of Lua to an R server via Redis. You might even
be able to hook directly into the R shared library from Lua with Lua's
C API, or embed Lua in R using the R C API. There are lots of ways to
skin this cat and very little reason to re-invent wheels.

-- 
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://j.mp/QCsXOr

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?