Lua as a data description language

[Dimitris Papavasiliou <jimmyp@...>]

Most games or game-like applications (i.e. applications that try to
model a world) I've seen (well all of them actually) seem to only use
Lua as a configuration/scripting language. It's great for this purpose,
no doubt about it, but after using Lua for a while in developing a game
engine, I thought: "why do I keep writing code to parse files, be they
binary or ascii, my own format or standard formats? Lua can do It all
and much better". A simple example:
Say you want to load a 3d model. Now loading 3ds files or others like
many people do is in my opininon completely lame as they've not been
designed for the purpose and introduce a lot of bloat, instability and
bugs. Writing your own simple format with just what you need is much
better but still, consider this: write a function called say model and
accepting a table with model data and returning some sort of model
object(like a userdata). Then in a file put this:
return model {
   vertices = {1.0, 2.0, 3.0, 4.0, 5.0, ...};
   indices = {1, 2, 3, 4, 5, ...};
};

then from an other script:
m = dofile "scripts/models/foo.model"
or
surface.model = dofile "scripts/models/foomodel.lua"

Again I thought to myself: "yay! Never will I need to write another line
of parsing code again. Long live Lua!". And I was right. This approach
gives you:
1) One consistent, clean, and aesthetic (IMHO) language for all your
data files.
2) "All your data files" can include images/maps, models, sounds, whatever.
3) No more parser writing (just converter writing :).
4) ASCII and/or binary files. Just do a $luac
scripts/models/foomodel.lua to get a binary file that will work just as
well as the ASCII script.
5) Propably others which I forget to mention since I now take them for
granted.

Well that's enough though-sharing for one night, let's get to the
bitching part. I have come across one problem with this approach (as
opposed to binary file formats). The fact that Lua just knows about one
kind of number (float, double or whatever you set lua_Number to) can and
will lead to space overhead. Consider this: Say you compile with
lua_Number set to float and you have the above model definiton. Vertices
which includes spatial and texture coordinates and normal vectors which
all need to be floats (in most cases) is no problem but for indices you
can usually get away with just shorts. This introduces a 2 byte/index
overhead. This again is not that big a deal since indices are relatively
few but if you try to store 4 RGBA values per pixel for an image map
where each value needs just be a char(a byte-sized integer that is) as
floats then you get 4 times larger image files. This is more serious
although it should be noted that the only problem is with file size and
hence resource loading time (once you get the data from lua you can cast
it to whatever type you want so it won't matter how lua stored it).
One way to "solve" the problem is to use strings instead of tables.
These strings will contain binary C arrays of floats, shorts or bytes
instead of Lua tables. Then the above example would look like this:

return model {
   vertices = "\123\034\167\123 ..."
   indices = "\132\056\032 ..."
}

This actaully simplifies the needed Lua code but I don't consider this a
solution as the whole idea is tha you can use a high level
machine-independent data description language whereas this is just like
reading a binary file but using lua to do the dirty parsing work. It
still is much better than parsing a binary yourself though, and just as
efficient.

Now I was wondering: since one of Lua's purposes is data-description.
And since it is logical to assume that some datasets can be large and
needing double precision here but only 8bit integer precision there, it
would be useful for Lua (and for me) to somehow be able to avoid wasting
space by storing everything as doubles. I haven't found a solution but I
do know that if a good one existis it:
1) will have to be implemented inside the Lua core as the problem is the
bytecode that the compiler creates: eg t = {1,2,3,4} will compile to
4*sizeof(double) bytes for the data and there's nothing you can do about
it using Lua code.
2) should not mess with Lua's current syntax. That is, solutions like
float = 4.0f or int = 2i or intv = {1, 2, 3, 4}i might work, but they
would introduce complexity into lua that would in my opinion definately
not be worth it.
I doubt that anything can be done about this because of the way Lua
handles numbers and number vectors but maybe I'm wrong. Any ideas on any
of the above subjects are welcome.

Dimitris P.