Re: [PROPOSAL 5.4] alternate way of format/pack/unpack (in the string library)

[nobody <nobody+lua-list@...>]

On 2017-05-27 15:01, François Perrad wrote:
> My main troubles with minilanguage used by `format` and
> `pack/unpack` are:

> - don't allow user extension

It is certainly feasible to use string.(format/pack/unpack) as
"low-level libraries", in the case of format and pack by formatting
single components (e.g. "%+.3f" or "I8") and `table.concat`ing the
results and in the case of unpack by supplying the optional start offset
and chaining through.  You can build just about any abstraction on top
of that and then extend it according to your tastes.

In their current form, they even permit optimizations by combining
several operations into a single format string – something that would
not (necessarily) be possible if the whole thing was delivered as a
bunch of separate functions and/or with an opaque type (like lpeg).

If no one has done this yet, it's not because it's hard (it's stupid /
boring boilerplate-y code), but simply because it's not needed.

(How do you (want to) use string.format/pack/unpack?  What I usually do
once I have my compound types is to define how to :prettyprint or
:(de)serialize them (or pretty( as_type, value ) / (de)serialize(
[type], value )) and then use that throughout.  I use format/pack/unpack
only at the lowest level (where primitive types are involved), anything
above that just needs to know that whatever it got can be pretty printed
/ serialized.  With that approach, I've never had the need to extend the
low-level format/pack/unpack building blocks.[1])

> - low-level is not exposed/accessible with a Lua API

Because they're not needed in C.  Instead of string.format you have
((a)s|f)printf, which works essentially the same way.  Instead of
pack/unpack, you can simply cast, define structs and/or apply
(hton|ntoh)(s|l).  Particularly the struct version I'd call _more_
convenient than pack/unpack.

-- nobody


[1] One thing that I did miss was an option for string.unpack to
aggregate results into a table.  But that's the only useful extension 
that I can think of.  Sean's example of (simplified)

x1, x2, x3, …, xn = string.unpack("Ix Iy Iz …", s)

is a symptom of the same shortcoming.  Maybe permitting

  foo, bar = string.unpack("I4 { I2 I8 I8 }", s)

as a shortcut for

  local x1, x2, x3, x4, x5 = string.unpack("I4 { I2 I8 I8 }", s)
  foo, bar = x1, { x2, x3, x4, x5 }

maybe even with "{ [some_valid_lua_identifier] I2 [x] d [y] d }" for

  local x1, x2, x3 = string.unpack("I2 d d", s)
  return { some_valid_lua_identifier = x1, x = x2, y = x3 }

possibly with "*5 pat" (read `pat` 5 times) or even (potentially too 
powerful?) "* I4 pat" (read `pat` (whatever the I4 at that position 
says) times).

That extension would permit *really* compact binary un/parsers that go 
directly from Lua tables to bytes and back.  But what I'm doing right 
now works fine, too.  Still, I'll keep experimenting with that some more…