Re: need help: some features of Lua don't fit well together

[Jerome Vuarand <jerome.vuarand@...>]

2015-04-14 16:37 GMT+01:00 Konstantin Osipov <kostja.osipov@gmail.com>:
> We're a database embedded into Lua, and struggling with coercing Lua
> type system into the data model we adopted.
>
> The data model of the database is a set of lists: every object
> (row) is a list of fields, a field can have arbitrary type, i.e.
> it can be a scalar or another list/map.
>
> Sometimes we need to automatically convert what a Lua function
> returns to a data set, e.g. to save it in the database.
>
> So we need a simple and consistent conversion scheme between
> various Lua returns and the data model.
>
> For example:
>
> return -> [] (an empty set of lists)
> return scalar -> [[scalar]] (a set with 1 list, the list has 1 element)
> return { 1, 2, 3 } -> [[1,2,3]] (a set with 1 list, the list has 3
> elements).
>
> The task looks simple at first, but any simple conversion scheme
> breaks on maps and multiple returns.
>
> Since there is no easy way to distinguish between a Lua map and an
> array, we have to traverse each argument of conversion, and check that
> its keys are numeric. If all keys are numeric, keys start with 1,
> and there are no gaps in keys, it's a list. Otherwise, it's a map and
> it must be treated the same way as a scalar - become a list field.
>
> This check is very inefficient, especially on large returns,
> and there seems to be no way to optimize it.
>
> With multiple returns, to put it simply, there is a lot of
> confusion.
>
> Should we treat each value of the multiple return as a field or as a list?
> Is return with 0 or 1 values a sub-case of returns with many
> values?
>
> If we assume that each value of the multiple return is a list, and together
> they form a set, then we can't return sets larger than ~8000 elements,
> that's about the limit for Lua stack size (at least this is the
> case in LuaJIT, please correct me if plain Lua is differnt).
>
> As long as we need a way to return a set which contains many lists, a
> special case is needed for very large sets.
>
> So the final (and best we could do) set of rules became like this:
>
> function convert_stack_to_set(...)
>     set = {}
>     n = 1
>     if the stack is 1 element and it is a Lua table and
>         it has only contiguous numeric keys, and keys start with 1:
>
>         for each table value do
>             set[n] = convert_value_to_list(value)
>             n = n + 1
>         end
>     else
>         for each stack value do
>             set[n] = convert_value_to_list(value)
>             n = n + 1
>         end
>     end
>     return set
> end
>
> function convert_value_to_list(value)
>     if value is a Lua table and it has only numeric keys, keys start with 1:
>         return value
>     else
>         return { value }
>     end
> end
>
> Bottom line, if we had a quick way to tell apart a Lua table from an array,
> or if multiple return object was coercible to a Lua table or array,
> there would be a scheme that worked well and could be easy to understand.
> The scheme that we came up with requires full traversal of the Lua
> stack before a decision is made and it doesn't seem to be easy to
> understand.
> Any help on a better set of rules and on optimization of the functions
> above is greatly appreciated.

Why do you need to distinguish the case when a Lua table is actually
an array? If you can store an arbitrary map, Lua arrays are just a
special case of that. In other words:

function convert_stack_to_set(...)
    local set = {}
    local n = 1
    for each stack value do
        set[n] = value
        n = n + 1
    end
    set.n = n -- if you care about trailing nils
    return set
end