Re: Possible improvements to the LNUM patch?

[David Given <dg@...>]

glen@worstell.com wrote:
[...]
> You will not have accuracy problems until you have a great many
> transactions. I wrote a quick lua program to see what the worst case would
> be, and it will never happen with my program unless you have (I forget how
> many - an extremely large number) of transactions.

It depends very much what you do. The errors show up immediately when
rounding edge cases.

70 cents + 5% == 0.70 * 1.05 = 0.735, which should round up to 0.74, but:

function round(n) return string.format("%.2f", n) end
print(round(0.70 * 1.05)) -> "0.73"

Likewise:

print(round(1.30 * 1.05)) -> "1.37" (but should be 1.36)

There are other problems, too; consider 1.0 % 0.1. The result should be
0, right? Try it:

print(1.0 % 0.1) -> "0.1"

All these operations and edge cases actually have *defined* behaviour in
the financial world. I gather it's actually *illegal* to use floating
point for some applications...

Here's a good explanation, although it is in Fortran:

http://h71000.www7.hp.com/wizard/wiz_8846.html

-- 
┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
│ "I have always wished for my computer to be as easy to use as my
│ telephone; my wish has come true because I can no longer figure out
│ how to use my telephone." --- Bjarne Stroustrup

Attachment: signature.asc
Description: OpenPGP digital signature