$ lua
Lua 5.4.2 Copyright (C) 1994-2020 Lua.org, PUC-Rio
> nan = 0/0
> t = {nan, nan, 20, 10}
> table.sort(t)
> print(table.concat(t, ", "))
-nan, 20, -nan, 10
Garbage in, garbage out. Every sort function I've ever seen has undefined behavior when you introduce NaNs, because a < b and b < a are both false when either is a NaN, so the sort function gets confused.
Don't feed NaNs to a sort function, or if you must, define your own comparison function that detects NaNs and treats them specially (e.g by designating them as less than any other value).