My 2 cents idea:
LuaOS would be built on top of an existing micro-kernel OS like (open source http://www.minix3.org/, not your grandfather Minix).
As most of you know, a micro-kernel OS has just a few functions running in kernel mode (e.g: clock,system interruptions, message passing, ...) and all other OS components run in application mode (e.g: file system, process schedulder, networking...). This means this means you can deploy this kind of OS whithout the unnecessary components as well as you can replace those components with the implementation of your choice (any ideas ? Lua ?).
As such, our micro-kernel based LuaOS should run only Lua VM based sealed processes (SIP or software isolated processes, as in Singularity: http://research.microsoft.com/apps/pubs/?id=69431).
I know that more simple to say as to code.