Re: Serialized test data in table format?

[Russell Haley <russ.haley@...>]

On Thu, Sep 28, 2017 at 2:51 PM, Sean Conner <sean@conman.org> wrote:
> It was thus said that the Great Russell Haley once stated:
>> Hi,
>>
>> I'm starting some testing on my lua-persist library that serializes
>> tables into lmdb. I suck at creating test data and keep getting
>> stalled out on the task. I thought I'd ask if anyone has some
>> serialized tables they could offer for my testing?
>>
>> One problem so far is that this thing is blazing fast. I had one set
>> of data that was around 6 MB and the system didn't even blink. If
>> anyone can offer 50-100MB of data (or more!) I'd be REALLY grateful.
>
>   At one time, I had an 83MB Lua script that generated a single file [1] but
> alas, that is long gone.  I just now generated a 44MB Lua script by crawling
> over some directories on my computer [2]:
>
> local dump  = require "org.conman.table".dump
> local fsys  = require "org.conman.fsys"
>
> local start = arg[1] or os.getenv("HOME")
>
> local function generate(path)
>   local data    = {}
>   local folders = {}
>   local oldcwd,err  = fsys.getcwd()
>
>   fsys.chdir(path)
>   local dir = fsys.opendir()
>
>   while true do
>     local entry = dir:next()
>     if not entry then break end
>
>     local info = fsys.stat(entry)
>     if info then
>       if info.type == 'file' then
>         data[entry] = info
>       elseif info.type == 'dir' then
>         table.insert(folders,entry)
>       end
>     end
>   end
>
>   dir:__gc() -- erm ... yea, kinda need this
>
>   for _,name in ipairs(folders) do
>     data[name] = generate(name)
>   end
>
>   fsys.chdir(oldcwd)
>   return data
> end
>
> local x = generate(start)
> dump("x",x)
>
>   If you don't feel like install my modules [3] you can probably get
> something similar with lposix or luafilesystem or something.
>
>   -spc
>
> [1]     http://lua-users.org/lists/lua-l/2009-11/msg00654.html
>
> [2]     Found a resource leak (well, kind of, the GC doesn't kick in as
>         often as it should) dealing with reading directories.  Ah, large
>         data ...  how well you find bugs!
>
> [3]     https://github.com/spc476/lua-conmanorg

I had to man-handle your makefile to build on freebsd and cut out the
net module due to errors (https://pastebin.com/yHARr5Gc), but I got it
to compile and install. However, there is no conman.table. I looked
and I found table.lua in the repo, but not in the installed directory.
I copied table.lua over and replaced your require. The final output
was a "bus error".  I didn't see anything immediately and draw the
line at going through your C code. :P

I hope to get to play with conman a little later. Your cbor stuff sounds neat.

Russ