Re: string.pack with bit resolution

[Sean Conner <sean@...>]

It was thus said that the Great bil til once stated:
> Hi Sean,
> thank you for your answer.
> 
> Sorry for my ironic start, this was ironic, I have to admit.
> 
> Concerning the "wicked 0x80" problem, you maybe program mainly on large
> systems - in case of large ints 4byte or more you hardly ever reach such
> limit. But if you program controllers which often use 1byte for a number,
> then this really can be very dangerous, believe me.

  I programmed nearly exclusively in assembly language (6809, 8088, 68000)
from 1985 to about 1995, so I am aware that a 16bit limit can be reached
quite readily, but again, I don't ever recall $80 (as a signed value) ever
being an issue for me.

> My proposal for this # and ^ is only meant as request TWO (and inferior) ...
> if you do not like this, then also fine, no problem. Just please at least
> add the bits. (and n1..7 already very fine...).
> 
> Just I think this # and ^ concept would increase the flexibility of this
> pack/unpack functions tremendously, and it will NOT lead to much further
> programming overhead in C, I am quite sure.
  
  And I think you are underestimating the complexities involved.  One format
I've been working with recently has the following 8-bits defined as:

	0 nn FFFFF
	1 nn i oooo

  That is, if the most significant bit is 0, then the last five bits is a
5-bit SIGNED value (-16 to 15), whereas if the most significant bit is 1,
bit 4 has a distince meaning, and the remain 4 bits define an operation,
with the 'nn' field (a 2-bit unsigned field) having mostly the same meaning
across both (but in the second case, 'nn' is a "don't care" for a few cases
of 'oooo'). Could I expect this to handed with a "bit formating langauge"?

  If no such distinctions are made, then I suppose I could live with:

	x nn zzzzz

and further parse the 'zzzzz' field depending upon the 'x' field but at this
point, I don't think I'm any better off than doing:

	local n = byte & 0x60
	if byte & 0x80 == 0x80 then
	  local i = byte & 0x10
	  local o = byte & 0x0F
	  -- handle this case
	else
	  local f = byte & 0x1F
	  if f > 15 then
            f = (-1 & ~0x1F) | f
            -- handle this case
	  end
	end

  Some other questions---assume we have "q[n]" meaning "an unsigned
collection of n bits (where n by default is 1).  What happens in the
following cases?

	x = string.pack("q",1) -- how many bytes is x? what should x be?
	x = string.pack("q17",5)
	x = string.pack("q17",-12)
	x = string.pack("q17q19q23",1,2,3)

> Of course I principally COULD program this myself. But it would be a very
> large advantage, if such a format is uniquily defined by somebody with
> "wider fan group", like e. g. lua ... then to communicate such format
> strings to other people, you just could specify "format string in lua-style"
> - would be very nice and useful in my point of view.

  You need to first convince the "wider fan group" first.

> Just it really has to be unique, and my side remarks concerning n, j and J
> ARE important... . It must clearly be defined such, that no problem to pack
> e. g. on a lua 64bit device, and then unpack on lua 32bit device and vc vs
> ... otherwise this pack/unpack does not make too much sense. 

  I can do that now.  The following binary format (it's the main ZIP file
header, by the way):

      end of central dir signature    4 bytes  (0x06054b50)
      number of this disk             2 bytes              
      number of the disk with the            
      start of the central directory  2 bytes
      total number of entries in the         
      central directory on this disk  2 bytes
      total number of entries in             
      the central directory           2 bytes
      size of the central directory   4 bytes
      offset of start of central             
      directory with respect to 
      the starting disk number        4 bytes
      .ZIP file comment length        2 bytes
      .ZIP file comment       (variable size)

is

	"!1<c4 I2 I2 I2 I2 I4 I4 s2"

  That will work to both pack and unpack that structure.

> I really think that these add's to pack and unpack would be easy to
> implement.

  The code is there.  It should be easy to add the functionality and release
it so people can bang on it.  I mean, even *IF* the Lua core team wanted to
do this, we won't see it until the next major release of Lua after 5.4
(which if history is any indication, won't be for a few years at least).
  
> and before no agreement in the community, it makes no sense to start
> programming ...

  That's a self-defeating way of looking at it.  If I find something
lacking and the maintainers aren't willing to make a change, I'll jump in
and do it myself---maybe they'll accept it, maybe not.  But at least I'm
solving my immediate issue.

> (as a main goal of these format definition clearly must be to transfer data
> in uniquely described form from one system to another, at least from lua
> 32bit to lua 64bit and vc vs - I hope you completely agree with this at
> least).

  Yes.

  -spc