sorry, but I don't quite get your problem:
By "locking up the GUI", do you mean the prompt window (or maybe a
modal dialog) locks its parent/owener window? If so, I don't think
there is problem, since in all window system a modal dialog is
supposed to take all the event processing work for the parent window.
here are some tips I hope are helpful.
the GUI is locked up because some event/message callback does not
return quickly to the event loop, so no new events can be dispatched.
you should divide the long event callback into shorter parts: you may
use coroutines to achieve this, or you may turn the big event into
many smaller events.
and you should be aware that the `plain' Lua 5.1 does not allow
coroutine to yield across a C function boundary, but the luajit (or
the coco patch with it, see the website of luajit) does.
while Lua 5.2 support yieldable C function, you have to manually deal
with the continous part.