• Subject: Hardware trap while casting large doubles to int on PowerPC P2020
• From: Thomas Jericke <tjericke@...>
• Date: Fri, 07 Mar 2014 08:17:07 +0000

On one of our targets a PowerPc P2020 we accountered a problem with the casting from double to int. The instruction that casts from double to in checks for the size of the number and traps on numbers that don't fit the int.
```
This problem was encountered while indexing tables with numbers.

const TValue *luaH_get (Table *t, const TValue *key) {
switch (ttypenv(key)) {
case LUA_TNIL: return luaO_nilobject;
case LUA_TSTRING: return luaH_getstr(t, rawtsvalue(key));
case LUA_TNUMBER: {
int k;
lua_Number n = nvalue(key);
lua_number2int(k, n);
...
}

The probem is the lua_number2int(k, n) in cases n > INT_MAX

I use the default definition which is:
/* the following definitions always work, but may be slow */

#define lua_number2int(i,n)    ((i)=(int)(n))

So much about the "always work" :-)

```
Now I looked into the C rationale and was not really sure if this is a non-standard behaviour, the standard says that in a cast to int the result is not defined in this case, but it doesn't say that the behaviour is undefined.
```
```
Anyway I thought a a solution that really always works is to define number2int as #define lua_number2int(i,n) ((i)= n > ((lua_number)INT_MAX ? INT_MAX : (n < ((lua_number)INT_MIN) ? INT_MIN : ((int)(n)))
```
So it n is > INT_MAX I just assign INT_MAX the same for negative numbers.
```
I am not sure if this is the correct behaivour that Lua would expect from lua_number2int in this cases. Maybe someone of the team can say if this works in all cases. I also assume that lua_number2integer and lua_number2unsigned would have to be redefined in a similar way.
```
```
For our project I now switched to the LUA_IEEE754TRICK for our floating point targets and this works on the P2020.
```
--
Thomas

```

• Follow-Ups: