Couple of things…
— In the main “for” loop you are adding a stack item for each loop iteration… this will overflow the stack if “size” is big enough. You need to pop the integer off the stack after assign it to array[I].
— Although its unlikely to happen in the real-world, you should ALWAYS check that calloc() does not return NULL. If it does, your function will crash the app with a pointer exception.
— I dont know how critical performance is here, but allocating a C array dynamically for small blocks is pretty wasteful. If it were me I would have a small array available on the stack, and use that if “size” were small (say, 16 or less), only doing the alloc/free if size were above this threshold.
— You might want to look at the function lua_geti(), which is a cleaner way to get a table item using an C integer index (easier to use than lua_gettable().