Re: [ANN] Lua 5.2.0 (work1) now available

[Mike Pall <mikelu-1001@...>]

Roberto Ierusalimschy wrote:
> > Summary:
> > - Inputs to bit operations need to be converted in a consistent and
> >   platform-independent way, respecting modular arithmetic requirements.
> 
> We would like to do that. But we do not know a reasonable way to do it
> in ANSI C89.

You can deal with it the same way you deal with popen(): make it
conditional, but enable it by default in this case (*).

The bit library needs to be made conditional, anyway: it should be
disabled for 32 bit floats, since the semantics just don't make
any sense there (loss of precision).

(*) Lua BitOp works on 16, 32 and 64 bit platforms with lua_Number
types IEEE 754 double, int32_t or int64_t. I've not found a single
OS/CPU/compiler combination that doesn't like it.

> > - Outputs of bit operations need to be signed.
> 
> We still dissagree. I cannot see how bit.not(0) could not be equal
> to 0xffffffff. Output of bit operations should be equal to the
> corresponding hexa constants.

The fact that hex literals have platform-dependent values is
unfortunate (but not easy to fix). Using this as the rationale to
add even more platform dependent behavior does not seem right.

After all, bit operations are used with decimal literals, too.
Results are passed around to other (signed) arithmetic operations.
And of course to C functions. These expect a signed 32 bit integer
in most places, so we better pass it to them as such.

> In C, if you get the result of
> a bit operation as a lua_Number and cast it to (unsigned int), you get
> consistent results with different types of lua_Number.

It's unlikely that all C embedders will change all of their C
bindings, just in case someone might pass the result of a bit
operation. And we all know that unsigned conversions can be slow,
so everyone will avoid it and will produce unportable code instead.

It makes far more sense to use signed numbers everywhere and keep
all the unsigned mess out of Lua. The instinctive reaction to use
unsigned numbers is a reflex trained by years of C coding, which
reflects some fundamental design choices for C. But it has not been
a design choice for Lua and I strongly advise not to make it one.

> > - Shift and rotate instructions need to be explicitly named.
> 
> We were already considering changing that. (But both directions will
> treat a negative shift as "shift in the other direction".)

If you have directional shifts, then special casing negative
shifts is superfluous. Nobody has asked for that feature and not a
single use case has been presented for it.

[About the FP representation example: I've picked apart FP numbers
with bit operations in the past. But 1) it uses a biased exponent
and not a signed one and 2) there is no need for a bidirectional
shift. This use case rather asks for a masked shift count!]

> > - Shift counts should be masked by the bit width.
> 
> Not convinced (see below).

You do not present any related arguments below.

I repeat my arguments for masking the shift counts:
- It enforces defined behavior.
- It's based on actual needs of users.
- It simplifies a majority of actual use cases.
- It works consistently for *all* shift or rotate instructions.
- It's consistent with the input conversions for other bit op
  arguments that respect the requirements of modular arithmetics.
- Performance is better that way, too.

> > - Arithmetic right shift is not optional.
> 
> Not sure.

You do not present any arguments. Actual use cases show that there
is a need. Existing bit libraries offer this functionality for
that reason, too.

> > - Ignoring the naming conventions defined by the most popular
> >   existing libraries is unwise, because it forces all users to
> >   rewrite their code (FYI: BitOp inherited the names from lbitlib).
> 
> That was a confusion. We should use the same names.

Here's the list for BitOp (which is a superset of lbitlib):

y = bit.tobit(x)
y = bit.bnot(x)
y = bit.bswap(x)
y = bit.band(x1 [,x2...])
y = bit.bor(x1 [,x2...])
y = bit.bxor(x1 [,x2...])
y = bit.lshift(x, count)
y = bit.rshift(x, count)
y = bit.arshift(x, count)
y = bit.rol(x, count)
y = bit.ror(x, count)
y = bit.tohex(x [,+-digits])

> When discussing language design, performance is always present. But
> we do not think that, in a scripting language, such as Lua, we should
> consider a change in an API just to avoid an extra conditional or a jump
> misprediction.

This makes it sound as though a "scripting language" is some
inferior creature that doesn't deserve as much attention to its
design. And Lua is rarely used in the way that Perl or Bash is, so
the name is undeserved as well.

As we all know, Lua is used because it's a dynamic language where
it's easy to write elegant, consistent and platform-independent
code with. And it owes much of it's popularity to its superior
performance. These are in fact its main selling points.

I cannot see how it's deemed acceptable to compromise on these
values, especially when there is a solution available that
conserves all of these properties.

If we use the "it's a scripting language, it can be slow" excuse
too often, we end up with Python. If we use the "it's a scripting
language, API design doesn't matter" excuse, we end up with PHP.
I sincerely hope I'll never see that day.

> For LuaJIT, I am sure it will be able to eliminate
> these extra tests in most cases.

Unfortunately not. E.g. variable shift counts are common and need
extra range checks that cannot be eliminated. The use of unsigned
results is not feasible at all since it doesn't interact well with
the double<->int32 duality of numbers that's used everywhere.

--Mike