Re: Formatting numbers without precision loss

[Hartmut Henkel <hartmut_henkel@...>]

On Tue, 20 Jun 2023, Thijs Schreijer wrote:

> Hey list,
>
> I’m running into the following issue (minimal example of a problem in a JSONschema validator):
>
> local maximum = 9007199254740991
> local code = "return function(val) return (val <= "..tostring(maximum)..") end"
> local func = load(code)()
> print(code)
> print(func(maximum))
> print(func(maximum+1))
>
> Results in:
> return function(val) return (val <= 9.007199254741e+15) end
> true
> true
>
> The second “true” is wrong, since it actually is 1 greater than the maximum.
> Because of the exponential notation used in the generated code, precision is lost, causing this mistake.
>
> Trying again, using a float formatting argument;
>
> local maximum = 9007199254740991
> local code = "return function(val) return (val <= "..("%.0f"):format(maximum)..") end"
> local func = load(code)()
> print(code)
> print(func(maximum))
> print(func(maximum+1))
>
> Results in:
> return function(val) return (val <= 9007199254740991) end
> true
> false
>
> Works. (I also tried “%d” instead of “%.0f”, but that rolled-over to a negative number)
>
> Now all of this is probably highly system/compile-time-config dependent. I get that.
> But keeping in mind that “value” that is passed in is represented in the system as a proper value, how can I format the number in the code such that does not loose any precision in it’s journey from; number => string => lua-parser => number ?
>
> Not just for ints but also floats? (And independent of Lua version, above was on PuC-Rio Lua 5.1)
>
> Any help is greatly appreciated.

fwiw, with Lua compiled with

#define LUA_FLOAT_DEFAULT LUA_FLOAT_LONGDOUBLE

in luaconf.h, i get:

hahe@hahepc1:~$ lua xx.lua
return function(val) return (val <= 9007199254740991) end
true
false

Best Regards
Hartmut

(I would love to see Lua optionally using libquadmath.)