So you're right, it's not thousands of times slower. It is slower by an order of magnitude though, which could be non-trivial when the whole system and everything anyone might ever build on top of it depends on the stack for literally *everything*. I think insert/remove is less clear than array notation anyway, but that's maybe personal preference more than anything.
Yes I would not use insert/remove if I needed performance, I was assuming you would use your own size tracking and use direct indexing, which of course avoids the need for Lua (or you) to use the (slow) ‘#’ operator. Also, if you track the array size yourself, you can also use ‘nil’ as a valid value on the stack.