If you need to perform complex comparaison, you should avoid doing that in the comparaison functions itself, but should prepare an accessory table of precomputed sort keys, that can be very quickly compared in the custom function and sort only that secondary table ascan index you can use to then index the main table (which can remain unsorted).
The basic table:sort() function does not has any integrated support to generate sort keys, but you may use the custom comparaison function to generate sort keys on the fly for thectwo objects to compare and cache them for further comparaison with other sort keys for other rows of the table.