Re: Dump large tables quickly

[Niccolo Medici <niccolomedici@...>]

On 7/3/18, Ravi Joshi <ravi.joshi53@yahoo.com> wrote:
>
> my_table = {p = {11, 22, 33, 44}, q = {0.12, 0.23, 0.34, 0.45, 0.56}}
> require "savetable"
> mytask.do_it(my_table)
>
> Please note that in the above code [...] in reality,
> the table contains two 1-dimensional arrays [...]

Tip: in C you can write just the function that serializes the
"primitive" array. If such arrays are contained inside more complex
tables, then you can provide a utility function, writen in Lua (as
it's easier), that serializes the whole table (I sent you an email
with a rock that demonstrates how to bundle Lua code together with
your C code). You can detect a "primitive" array using heuristics, for
example, a table whose #1 element is a number.

>
> (1) savetable.c: https://pastebin.com/sBWA3uaM

It'd be more useful if your function wrote the data into a string and
returned it (string are binary-safe). Then the programmer will be free
to use it however she wants.

>
> I am using Lua 5.1 (sorry, I said 5.2 previously
> by mistake)

There are small differecnes between the APIs. Test your rock with all
the versions you want to target. You can make your C code portable by
using "#ifdef LUA_VERSION < ..." (or whatever the name of the verison
def is called).

>
> I am looking for suggestions, to make the table serialization much faster.

The bottlneck is extracting the numbers from the table, and I guess
you can't do too much about this.

You can implemnt the array data-structure in C, and expose it as a Lua
userdata. You don't need much code for this. Then serialization should
be fast (as you'd store it in memory as a linear sequence of doubles).
It's hard for me to believe that nobody has done this already (numpy
anyone?). So, again: check in luarocks.org.

>
> on 64 bit Ubuntu PC.

BTW, if you want the serialized output to be portable among systems
you need to consider endianness (google it).

>
> (1) savetable.c: https://pastebin.com/sBWA3uaM

I have criticism for your C code, of course, but I used up all my time
already ;-) If people here don't comment, email me in a few days with
your updated code and I'll comment on it.

(BTW, your email ended up in my spam folder (I'm using gmail). Maybe
that's why you haven't got much replies. Things to try: don't send
HTML, don't top-post.)