Packing data (was Re: [PROPOSAL 5.4] alternate way of format/pack/unpack (in the string library))

[Sean Conner <sean@...>]

It was thus said that the Great tobias@justdreams.de once stated:
> 
> Quoting Sean Conner <sean@conman.org>:
> 
> >  I don't find the pack/unpack minilanguage all that bad, per se.  
> >  Lowercase
> >letters are signed quantities, uppercase unsigned and there's some 
> >mneumonic
> >meaning to the letters used.  But it can get silly (sample from an SMPP
> >parser):
> >
> >	result.service_type,
> >	result.source.addr_ton,
> >	result.source.addr_npi,
> >	result.source.addr,
> >	result.dest.addr_ton,
> >	result.dest.addr_npi,
> >	result.dest.addr,
> >	result.esm_class,
> >	result.protocol_id,
> >	result.prority,
> >	result.schedule_time,
> >	result.validity_period,
> >	result.registered_delivery,
> >	result.replace_if_present,
> >	result.data_coding,
> >	result.sm_default_msg_id,
> >	result.message =
> >	string.unpack(">z I1 I1 z I1 I1 z I1 I1 I1 z z I1 I1 I1 I1 
> >	s1",blob,13)
> >
> >  It was hard to debug, and the obvious solution:
> >
> >	result.service_type,pos    = string.unpack(">z",blob,pos)
> >	result.source.addr_ton,pos = string.unpack(">I1",blob,pos)
> >	result.source.addr_npi,pos = string.unpack(">I1",blob,pos)
> >	--- and so on
> 
> This is exactly what I'm trying to solve with lua-t library Pack module.  It
> doesn't support zero terminated strings yet but once it does your code could
> look like:
> 
> snppParser = Pack(
>    { service_type         = ">z" },
>    { source               = Pack(
>       { addr_ton                 = 'I1' },
>       { addr_npi                 = 'I1' },
>       { addr                     = 'z'  }
>    ) },
>    { dest                 = Pack(
>       { addr_ton                 = 'I1' },
>       { addr_npi                 = 'I1' },
>       { addr                     = 'z'  }
>    ) },
>    { esm_class            = 'I1' },
>    { protocol_id          = 'I1' },
>    { prority              = 'I1' },
>    { schedule_time        = 'z' },
>    { validity_period      = 'z' },
>    { registered_delivery  = 'I1' },
>    { replace_if_present   = 'I1' },
>    { data_coding          = 'I1' },
>    { sm_default_msg_id    = 'I1' },
>    { message              = 's1' }
> )
> 
> result = snppParser( blob )

  Actually, that's just one of several different packet types for SMPP, and
not included there are the optional parameters.  I'm reading your
documentation on Pack [1] and I'm not seeing anything that can actually
handle the optional part of the SUBMIT_SM packet (what I'm actually encoding
there).  You list:

	atomic		A single instance
	sequence	Multiple values in a given order
	array		Multiple values of the same type
	struct		Ordered collection of named values

  The optional parts of the SUBMIT_SM are a series of values that can appear
in any order.  Now, they all follow a similar format:

	tag	2 byte integer network byte order
	length	2 byte integer network byte order
	value	variable, based upon length

  For an even weirder example, we have a custom binary format.

	tag	1 or 3 bytes
	length	1 or 3 bytes
	value	variable length

  For the tag or length, if the byte is less than 255, then that's the
value; if it is 255, then the actual value is the next two bytes, in network
byte order.  The order of tags does not matter, *except* that tag value of 1
should always be first.

  CBOR [2] is similar---integer values can be 1 byte, 2 bytes, 3 bytes, 5
bytes or 9 bytes in size, and the type is encoded with the value in some
cases.

  Then there are protocols like IP, where some fields are only a few bits in
size, but it looks like you handle that somewhat.

  -spc

[1]	https://github.com/tobbik/lua-t/blob/master/docs/Pack.rst

[2]	Concise Binary Object Representation RFC-7049
	http://cbor.io/