Anyway, in other applications outside Lua, I've completely zapped the use of hashtables with a static large number of BUCKETS in the root page, replacing it by b-trees using much smaller pages with fixed size. You get the benefit of more compaction, a natural ordering for sequential access, fast random access, no longer any worst case of long collision lists, and much more efficient memory management. And you don't need to tweak into separating an array part and an hashed part, you get both simultaneously with pages that are small sorted arrays, and still the support for hashes when necessary. And data in the btree can also be compressed by not relegating the common prefixes in every node of the page.
That's what all modern databases do for their storage on indexed tables. B-trees are not so complicate, and they still maintain very good data locality. They are excellent for performance, and very economic in terms of storage space which is autoadaptative.
I' m convinced that all Lua tables could be implemented using a single b-tree, whose pages can also be stored in a single array, resizable on demand if more pages are needed.