Re: Lua 5.1.3 bug in string.byte

[Jürgen Hötzel <juergen@...>]

On Tue, Jun 03, 2008 at 01:37:03AM +0200, Mike Pall wrote:
> The range check logic in string.byte is broken:
> 
> $ lua -e 'for i=5,-10,-1 do print(i, string.byte("abc", i)) end'
> 5
> 4
> 3       99
> 2       98
> 1       97
> 0
> -1      99
> -2      98
> -3      97
> -4
> -5      97      98      99  <--- Unexpected results for -#str-2 and lower
> -6      97      98

I also have run into this bug. posrelat is called twice for the
default end index (optional parameter not supplied).

Also there is an invalid cast to size_t which changes a negative end
index to an positive value.

Jürgen

--- lstrlib.c.orig	2008-06-03 14:33:01.000000000 +0200
+++ lstrlib.c	2008-06-03 14:52:06.000000000 +0200
@@ -106,10 +106,10 @@
   size_t l;
   const char *s = luaL_checklstring(L, 1, &l);
   ptrdiff_t posi = posrelat(luaL_optinteger(L, 2, 1), l);
-  ptrdiff_t pose = posrelat(luaL_optinteger(L, 3, posi), l);
+  ptrdiff_t pose = posrelat(luaL_optinteger(L, 3, luaL_optinteger(L, 2, 1)), l);
   int n, i;
   if (posi <= 0) posi = 1;
-  if ((size_t)pose > l) pose = l;
+  if (pose > (ptrdiff_t)l) pose = l;
   if (posi > pose) return 0;  /* empty interval; return no values */
   n = (int)(pose -  posi + 1);
   if (posi + n <= pose)  /* overflow? */