string.unpack segfaulting due to integer overflow and mixed sign comparisons

[Sam Roberts <vieuxtech@...>]

lpack for 5.1 from:

http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack

testcase:

string.unpack("", "A"..0xffffffff, 2)

On linux i386 with 32-bit int and size_t.

Generally, the lack of checking for invalid args, like negative string
positions, makes me nervous. In particular,

lua -l pack -e 'print(string.unpack("", "b", -100))'

because int i = -100; size_t len = 0; if(i+1 > len) evaluates to
true... the -99 gets promoted to a very large size_t, so ends up being
larger than most string lengths.

Also, while (isdigit(*f)) N=10*N+(*f++)-'0'; will wrap N for large
enough repeat counts, though this should just result in confusion if
it wraps back into the 0 < N < len range, not segfaults.


fix:

Index: pack/lpack.c
===================================================================
--- pack/lpack.c        (revision 27854)
+++ pack/lpack.c        (working copy)
@@ -129,7 +129,7 @@
    case OP_STRING:
    {
     ++N;
-    if (i+N>len) goto done;
+    if (i+N < i || i+N>len) goto done;
     lua_pushlstring(L,s+i,N);
     i+=N;
     ++n;