A maximum string size is not enough, what is important is the total memory (used by all strings, if they are different and not canonicalized to the same immutable copy, but also all other objects).
You could as well run a string that creates a giant table with tons of different strings:
t = {} for i=1,1E100 do t[i] = tostring(i) end
You could as well create infinite recursion easily with a broken implementation of the factorial function:
function fac(n) return n < 2 and n or n * fac(n+1) end
(with the typo changing the correct - into a +)
Here, you'll exhaust the stack limit, and what happens at this step is unpredictable (it may be difficult to even send an error or force the current function to return nil, if it requires adding an additional stack frame for building an error object, or to call a debug.traceback, before unwinding the stack up to the topmost pcall() error handler.
As well pcall() may be called recursely infinitely :
function fac(n) if n < 2 return n local ok,ret=pcall(fac(n+1)) if ok then return n * ret else return nil end
This code still looks correct as it handles the error, but unwinding is limited, and nothing prevent that function to retry the failed pcall (and it will fail again):
function fac(n) if n < 2 return n repeat local ok,ret=pcall(fac(n+1)) if ok then return n * ret until ok nil end
As long as the fix changing the incorrect + into a - is not done, the program still breaks infinitely and insists in using all memory allowed for the standard call stack (and the pcall stack).
This does not seem to be a case of UB, but what happens when the stack can't grow is UB (and Lua has lot of difficulties to track and check the stack usage as the tracking code also depends on correct behavior of the internal C/C++ error handlers used by the Lua engine implementation, which itself also depends on error handlers in the OS itself).
Asserting if the quotas cannot be exhausted and failure recovery is possible and safe, requires writing and running stress tests as a routine task for programmaers. A very classical programming problem, not always easy to write for complex programs that fail in very specific or complex conditions hard to reproduce.
And many programmers to take the time to write correct stress tests (just like they often do not insert any assertion check in their code, except in rare places, thinking first in terms of practical performances for the most common cases they want or have to handle, taking time to write a program that will voluntarily bug is considered a waste of work time... until later someone reports a severe bug that will be harder to isolate later).
