bug in table.sort (lua 5.1.3)

[Tommy Pettersson <ptp@...>]

Hi,

I'm not a list subscriber. Well, I am now, but I don't think
I'll have time to read it.

I just wanted to submit a bug fix. It is very minor, and only
concerns the error reporting from Lua. But it sent me on a long
bug hunt that could have been much shorter.

Here's how to reproduce the bug:

  local t = { 1 }
  table.sort( { t, t, t, t }, function (a,b) return a[1] == b[1] end )

gives:

  lua: stdin:2: attempt to index local 'a' (a nil value)

but should give:

  lua: stdin:2: invalid order function for sorting


It's auxsort that calls lua_rawgeti with an "out of bound" index
and thus passes a nil value to the comparison function.

The fix:

diff -rN -u old-lua-5.1.3/src/ltablib.c new-lua-5.1.3/src/ltablib.c
--- old-lua-5.1.3/src/ltablib.c	2008-04-04 00:51:38.375673575 +0200
+++ new-lua-5.1.3/src/ltablib.c	2008-04-04 00:51:38.407673452 +0200
@@ -216,12 +216,12 @@
     for (;;) {  /* invariant: a[l..i] <= P <= a[j..u] */
       /* repeat ++i until a[i] >= P */
       while (lua_rawgeti(L, 1, ++i), sort_comp(L, -1, -2)) {
-        if (i>u) luaL_error(L, "invalid order function for sorting");
+        if (i>=u) luaL_error(L, "invalid order function for sorting");
         lua_pop(L, 1);  /* remove a[i] */
       }
       /* repeat --j until a[j] <= P */
       while (lua_rawgeti(L, 1, --j), sort_comp(L, -3, -1)) {
-        if (j<l) luaL_error(L, "invalid order function for sorting");
+        if (j<=l) luaL_error(L, "invalid order function for sorting");
         lua_pop(L, 1);  /* remove a[j] */
       }
       if (j<i) {


-- 
Tommy Pettersson <ptp@lysator.liu.se>