Re: Data transformation templates

[Alexander Gladysh <agladysh@...>]

On Wed, Jan 8, 2014 at 10:48 PM, Alexander Gladysh <agladysh@gmail.com> wrote:
> On Wed, Jan 8, 2014 at 5:52 PM, Javier Guerra Giraldez
> <javier@guerrag.com> wrote:
>> On Wed, Jan 8, 2014 at 8:16 AM, Alexander Gladysh <agladysh@gmail.com> wrote:
>>> I'd like to write this declaratively. Maybe like this:
>>>
>>> local result = T:map (data)
>>> {
>>>   sprite = "/img/" .. T(1);
>>>   x = T(2), y = T(3);
>>> }
>>
>> what's wrong with the classic map(t,f) transformation?
>>
>> function map(t,f)
>>     local o={}
>>     for i = 1,#t do
>>         o[i] = f(t[i])
>>     end
>>     return o
>> end
>>
>> local result = map(data, function(e) return {
>>     sprite = '/img/'..e[1],
>>     x = e[2], y = e[3]
>> })
>
> Well, my example is too simple, probably :-) I do not care about speed
> at all in the most of the relevant applications — they all are
> middleware tools, doing code (or rather data) generation.
>
> I need to do much more complex transformations. Imperative (or
> functional) code simply does not cut it — it is not readable.
>
> The point here is not the "stunt" with that fancy T syntax. The point
> is to make code declarative to enhance readability.
>
> Compare a {{mustache}} template with the same text, created via
> concatenations (or, ok, using ropes, doesn't matter) in imperative (or
> even functional) Lua code.
>
> Here is a more complex example:
>
> https://gist.github.com/agladysh/6f99595fc289fc28e251
>
> This is a part of a larger data transformation thing, maybe of 500 lines.
>
> Yes, the data format design is rather awful here — both for incoming
> data and for output. But one of the reasons of why it is awful is that
> it is too obscure and unreadable to start with. It is hard to keep the
> whole picture in the head when one has to organically grow the thing
> due to changing requirements while the time is pressing.

I'll give a bit more background here, maybe it'll help.

When designing a DSL-ized data-driven system (we have quite a few, and
find them very convenient and useful), I usually implement two levels
of data format.

First is the high-level format that strives to be nice and
human-friendly, and is filled with specialized syntax sugars and other
stuff to ease human's work. It is translated into a (usually) generic
and very machine-friendly low-level data format. [1]

A human codes in the high-level format, and an off-line middleware
tool translates it to the low-level format, doing all manners of the
validation and data transformations, and that low-level format data is
then fed to the on-line system.

Machine-friendliness of the low-level format allows us to keep the
implementation of the main system relatively simple and easy. At the
same time, high-level format is friendly to a human and allows to code
(or describe the data) in the DSL with relative ease.

Now I'm slowly approaching the point where the DSL itself can be
specified in a nice powerful and unified way. As I mentioned in my
last Workshop talk[2], I experiment with using FSM to describe the
(internal/embedded) DSL [3]. Current syntax[4] is rather awful and
badly-designed (that's a prototype, that's why!), but it is already,
IMHO, a significant step forward from an approach, described in my Lua
WS'11 talk[5] (which is, by itself, much better than the usual ad-hoc
way to build internal DSLs — IMO, at least). I believe that I see
where to go from that point, and the future is brighter than the
present :-)

Anyway, I'm successfully using this prototype approach in a small
middleware tool that I've built. Needless to say, it is doing a data
transformation from a human-friendly DSL-ized data to a set of
machine-friendly good old Lua tables. The tool also uses several more
data sources (specifically, a tool configuration file, a list of
available asset files and some tabular date in CSVs).

I came to a point where most of the implementation of the data
transformation behind the DSL is written using DSL FSM approach. There
are several important exceptions, though, where I have to write a
bunch of code like I did in the Gist linked above. (I could redesign
the language implementation to move data transformation inside the DSL
FSM description, but that is not the point here.)

The problem is that a significant part of data transformation code
does not belong neither to the low-level code of the tool itself, nor
to the DSL FSM implementation. Much of the end-data should be editable
by a power user, not DSL "system" programmer.

That is, in code like [6] I'd like to load a template from the file and write

return <user's template>:fill(<my data for each place-holder in the template>)

instead of

return
{
  huge bunch of functional-ish code
}

This, of course, was a problem before the DSL FSM approach, but it was
not so apparent, because everything else was coded in functional-ish
style. Now it stands out, and I'd like to solve it nicely.

It is not a big problem to actually implement a (sub-)DSL to allow
user to provide such templates. But I do not want to redesign the
wheel here and design something arbitrary — and, thus, I'm asking for
an existing solutions.

Hope that I'm clear enough this time :-) If not — I'll be happy to
clarify further :-)

Alexander.

[1] I've talked a bit about this approach in my Lua WS talks:

http://www.slideshare.net/agladysh/declarative-internal-dsls-in-lua-a-game-changing-experience
(slides 59 and 60)

http://www.slideshare.net/agladysh/luaws13-ag (slides 33 and 34,
probably best viewed along with the video — I hope I said something
useful there on that matter :-) )

Here both formats are written in DSL, more often second format is just
a set of plain old tables.

[2] http://www.slideshare.net/agladysh/luaws13-ag, slides 41–43.

[3] http://bit.ly/le-dsl-fsm

[4] https://github.com/logiceditor-com/le-dsl-fsm/blob/master/src/lua/dsl-fsm/util/path_mt.lua#L85

[5] http://www.slideshare.net/agladysh/declarative-internal-dsls-in-lua-a-game-changing-experience

[6] https://gist.github.com/agladysh/6f99595fc289fc28e251#file-gistfile1-lua-L39