although 63-bit integers are possible by using the sign bit of the NaN to distinguish them, and the loss of one bit from a 64-bit integer is not likely to be a problem for the majority of applications.
That's an incomplete (or false) statement:
- A NaN uses a fix value of the WHOLE exponent field (the same value also used by "infinites", which is ALL bits of the exponent field set to 1), except that infinites set the mantissa field to all zeroes and uses the sign bit, whereas NaN does not use the sign bit, but must have at least 1 bit set to non-zero in the mantissa field to distinguish it from infinites.
- This means that in a NaN packing you can encode exactly only integers as wide as the mantissa field minus + the sign bit, but also making sure that the mantissa field has at least one bit set to 1
- in addition, the integer value 0 will be represented by the floatting point value 0.0 (not using NaN packing) using a different exponent field set to 0. You can use positive values of the exponent field up to the size of the mantissa for encoding integers as well, meaning that NaN packing would be used only for integers above this range ONLY to add 1 extra bit of precision.
- So the final range of integers that are encoded exactly is limited to the size of the mantissa field plus 1 bit (the sign bit) or 2 bits (with NaN packing, but then you loose the possibility of encoding other datatypes).
Given that the exponent field is 11 bits and the mantissa field is 52 bits, the maximum range representable is 53 bits, NOT 63 bits.
NaN packing will be used for encoding all other types in the 52 bits of the mantissa field plus the sign bit: you'll have to make sure that 1 bit in the mantissa is set to 1 and you have 51 bits encodable: just set the highest bit for that (if it is zero, the value still represents an NaN, but then you can set the next highest bit to 1 to use the lowest 50bits of the mantissa)
How to encode Lua types (other than numbers) efficiently: just reserve the 4 highest bits of the mantissa (don't use the all-zeroes) for that and the sign bit; it always remains the 48 lowest bits in the mantissa for encoding values in each type (including for example the 6 bytes of short strings, provided that you've encoded the short string-length in the type field you've reserved at top of the mantissa; for example you can set the sign bit to indicate that you use the 48 bits for short strings; and otherwise clear the sign bit for all other types)
You can also have 48 bits for storing pointers (or integer handles if you need an external table of pointer without the pointer alignment restriction) for each type that needs them ("long" strings larger than 6 bytes, tables, [light]userdata, functions and cfunctions, threads/coroutines)...
Other simple types can be encoded exactly (nil, false, true, possibly other small types that your engine would like to handle faster internally, such as small integers if you don't want, at every arithmetic operation, to have to adjust the exponent field to normalize/shift the value or don't want to use the floatting point sign in 1-complement notation: reserving these patterns allows you to delay the renormalization to a floatting point before returning to Lua; this can be useful notably to implement a JIT compiler that would convert the Jua bytecode to native code where normalization is delayed and where you can just use native integer registers and the native integer arithmetic instruction set).
But in all cases, you CANNOT encode 63-bits integers, unless you've completely sacrificed the usable valid value range of floatting points and the possibility to keep a distinctive NaN value and two infinite values (this sacrifice will break many Lua apps). But 53-bit integers is safe and does not require any change in the Lua code, and ius compatible with CPU/FPU instruction sets, and also allows you to use a vary wide 48-bit addressing space for each reference type. And preserving all Lua type semantics is possible: NaN-packing in these conditions is not only safe, it is very fast, very efficient on all 64-bit platforms (for 32-bit platforms, it is more complicate to be efficient in all cases when using IEEE 32-bit floats, you can just improve some cases, and the packing scheme is harder to implement and test correctly, whereas is it trivial on 64-bit platforms).
