In a typed language this would be somewhat easier because you know which stack slots and array entries have to be visited to decrement the reference count, but in Lua this would introduce overhead in every function call, function return, every assignment, etc. etc. because you don't know where these reference counted object are referenced
from. You end up paying for the overhead of reference counting AND for the overhead of the regular GC with its write barriers.
I believe it would be a better idea to introduce something like Python's context managers ('with' clause) or C#'s 'using' clause, which will notify your object when you leave the function it was created in, even if you exit the function via an error/exception. This would have far less overhead, and no overhead if you don't use it. (I know this has been discussed before).
(Another problem with reference counting is that the decrement-reference-count operation can cause a recursive deallocation cascade that may take an arbitrarily long time. You can perform a lazy form of reference counting, but then you may as well use the current GC).
