Re: io.format (was: Reported to the Bloat Police: string.pack, .unpack and .packsize!)

[Sean Conner <sean@...>]

  When this first came up, I was wondering where you (John Hind) have been. 
Lua 5.3 was announced in July 2013, released in January 2015 and *NOW* you
realize the string library has been expanded?

  I think it's a bit late for that.

  Anyway ... 

It was thus said that the Great John Hind once stated:
> 
> My point here is that high level languages are supposed to be something
> you can learn and use without having to consult a reference guide all the
> time. Machine-code like terse arbitrary syntaxes embedded within languages
> spoil this party.

  That's one definition of a high level language, but far from the only one. 
Another one might be---a high level language allows the programmer to
express intent (what is to be done) rather than method (how it will be
done).  Here's one example---given:

	struct foo a;
	struct foo b;

we have

	memcpy(&a,&b,sizeof(struct foo));

vs.

	a = b;

K&R C only allowed the former; ANSI C now allows the later.  The second one
conveys the intent better (I want a to have the same contents as b) and
doesn't bury it in how it is done (memcpy()).

  By allowing the programmer to express intent, there are futher
opportunities for optimization (speed, memory usage, development time).

> The fact these have been inherited from C is not a good excuse - by this
> logic Lua is unnecessary and we should all just use C!
> 
> I cannot actually think of a better replacement for regular expression
> patterns (does not mean I have to like them!).

  LPeg (for a lot of reasons), but seeing how that's a separate package (for
now) it doesn't fall into this argument.

> String formatters and string literal escapes can both be replaced by
> string concatenation and conversion/production functions. string.char is
> already adequate for generating strings of non-printable bytes (although
> constants for the common control codes like CR and LF would be useful).
> tostring could easily be enhanced with optional parameters to control the
> fill width, number of significant digits etc.

  I'm not sure what your point is here.  Yes, this:

	syslog('emergency',"%s stat(%q) = %s",dbtype,conf.file,errno[err])

could be replaced with:

	syslog('emergency',dbtype .. ' stat("' .. conf.file .. '") = ' .. errno[err])

but if I had to do that, I think I would end up implementing string
formatters just to better express intent.  

> On the main point of my post, I was not suggesting string.pack etc. is
> useless, just that is should be a separate (optional) library. The
> situation could be recovered somewhat by moving them to a sub-library in a
> future version enabling them to be omitted without having to modify the
> string library source.

  I wouldn't mind string.pack() and string.unpack() in their own library,
but not because of "bloat", it would make it easier to backport to Lua
5.1/5.2.  I find string.pack() to be exremely useful for what I do (if I
were using Lua 5.3, which for various reasons, I'm not yet).

> The use cases posited have been:
> 
> Marshalling between multiple Lua threads: 
                      ^^^^^^^^^^^^^^^^^^^^

I think you mean multiple, separate Lua states here.  There is a function
(lua_xmove()) to move Lua values between Lua threads (coroutines).

> here you have control of both ends, so you do not need something this
> sophisticated. I would tend to serialise the data to Lua for this. In any
> case you will need some kind of multi-thread library as well, so the
> marshalling facilities may as well be included in that (or in a separate
> support library).

  One use case that comes up from time to time that you haven't covered is
moving data from one Lua state on one computer to another Lua state on
another computer (that may very well be a different architecture).  Yes,
networking will probably be needed, but you don't need a multi-threaded
library for this case.

> Reading/writing  proprietary binary file formats: This is surely a VERY
> specialised requirement again arguing for the facility to be in an
> optional library. As I pointed out in the OP, it is also quite legacy
> since most modern code uses structured text file formats. The bit-wise
> operators (which the Bloat Police quite rightly approved!) make it
> relatively easy to write this stuff in Lua anyway.

  I was unaware that ZIP files were a proprietary binary format.  I wonder
whose intellectual property I'm violating by decoding DNS packets or CBOR
formatted data.

  But we also get to the intent vs. implementation argument, as well as
optimizations.  In Lua 5.1, to read in a 32-bit value from network byte
order to native order, I have the following code:

	val = x:sub(1,1):byte() * 2^24
            + x:sub(2,2):byte() * 2^16
            + x:sub(3,3):byte() * 2^8
            + x:sub(4,4):byte()

Now, I could use the following for Lua 5.2:

	val = bit32.bor(
		bit32.lshift(x:sub(1,1):byte(),24),
		bit32.lshift(x:sub(2,2):byte(),16),
		bit32.lshift(x:sub(3,3):byte(),8),
		x:sub(4,4):byte()
	)

and frankly, I think the Lua 5.1 version is a bit clearer in intent (yes,
these could be made into functions to hide the grotty details).  Contrast
this with Lua 5.3:

	val = x:unpack(">i4")

  Even if this is a bit harder to read, I content it's easier to read than
the Lua 5.2 version, and it will be faster than either version as it's a
function call (string.unpack()) vs. 8 (5.1 plus 6 math operations in Lua) or
12 (5.2)) and it's pretty much *already* a function that's been encapsulated
for us.

> I must also point out that as currently implemented, we have a byte
> packing facility not a bit packing facility which restricts the use cases.
> You can pack more tightly than a structured text format could, but not as
> tightly as possible.

  Bit packing might cause even more bloat.  

  -spc (I'm just saying ... )