It's a hybrid merge-insertion sort. Running under LuaJIT, it's
pretty fast.
I only wrote this quickly and haven't given it intensive testing, but this sort function is pretty simple, short, stable and fast (only tested your function in vanilla Lua - it performed about 20% faster) - so without warranty, here's the code:
--- sorts arr in a stable manner via a simplified mergesort
-- the simplification is to avoid the dividing step and just start
-- merging arrays of size=1 (which are sorted by definition)
function msort_inplace(arr, goes_before)
local n = #arr
local min = math.min
local step = 1
local fn = goes_before or function(a,b) return a < b end