Re: Dump large tables quickly

[Ravi Joshi <ravi.joshi53@...>]

Thanks, Niccolo Medici. It was wonderful to hear your suggestion.

I extended your suggestion and made a sample Lua C binding. Below is the Lua code-

my_table = {p = {11, 22, 33, 44}, q = {0.12, 0.23, 0.34, 0.45, 0.56}}
require "savetable"
mytask.do_it(my_table)

Please note that in the above code for debugging purpose, I have defined two very small 1-dimensional arrays. However, in reality, the table contains two 1-dimensional arrays in which each array contains approximately 800,000 elements.

I am not putting the C code using Lua C bindings in this message since it can take many lines here. Instead, please check following Pastebin links to get the code-

(1) savetable.c: https://pastebin.com/sBWA3uaM
(2) wrapper.lua: https://pastebin.com/uC4vpacC

I am looking for suggestions, to make the table serialization much faster. I am using Lua 5.1 (sorry, I said 5.2 previously by mistake) on 64 bit Ubuntu PC. 

PS: I am new to Lua community, I couldn't find much help on the luarocks.org. Do you know any such library?

Thanks again

-
Regards
Ravi





On Sunday, 1 July, 2018, 4:49:01 PM GMT+9, Niccolo Medici <niccolomedici@gmail.com> wrote: 





On 6/30/18, Ravi Joshi <ravi.joshi53@yahoo.com> wrote:

> Hi all,
>
> I am using Lua 5.2. I have large tables (1-dimensional arrays) of size
> 800,000. I want to dump these tables quickly. I found an article on Wiki
> titled "Save Table To File" [1] and used it but found not up to the mark.
> The tables saved using this method, i.e., table.save( table , filename ) are
> shared in my DropBox here [2].
>
> Since my primary concern is speed, hence I am ready to adopt binary file
> serialization if such exists.
>
> References-
> [1] http://lua-users.org/wiki/SaveTableToFile
> [2] https://www.dropbox.com/s/efdfvgy1n4m0zbx/table.dump?dl=0



On my ancient computer, the following code takes 8.5 seconds:

    local t = assert(loadfile('table.dump'))()[1]

    for i = 1, #t do
      print(t[i], ",")
    end

And using this simple-minded C code, it takes 1.3 seconds:

    static int
    l_dump(lua_State * L)
    {
      int len;
      int idx;

      printf("{");
      len = lua_rawlen(L, -1);
      for (idx = 1; idx <= len; idx++) {
        lua_Number n;
        lua_rawgeti(L, -1, idx);
        printf("%g,\n", (double) lua_tonumber(L, -1));
        lua_pop(L, 1);
      }
      printf("}");

      return 0;
    }

That's my first try: I guess this code can be made faster (and you
should make it a luarock, and also make it portable among the Lua
versions). But first search in luarocks.org to see if somebody has
done this already.