Given that "s={}; getmetatable(s)" returns nil in Lua, it is ovbious that not all objects necessarily have a metatable. So you can set it, and can remove it as desired, provided that the "global" function setmetatable() visible in the current scope allows this change and was not overridden to do something
But here lua_setmetatable() is the C API which has no lexical scope in Lua, so it remains accessible as is: it is the global function...
The doc effectively says "Pops a table from stack" but could say "Pop a value from stack"
And then "sets it as the new metatable for..." could be "sets it as the new metatable (if the poped value is a table) or detach the current metatable (if the poped value is nil) for..."
I think it is obvious from the description of getmetatable and the fact that this functions allows overwriting the metatable property of the object value which currently references another table or still no table at all).
But it would make the sentence more difficult to interpret.
What is missing however is that when the new metatable replaces an existing metatable, the former metatable gets dereferenced, so its counter of references will be decremented, and the former metatable could become garbage collectable at any time after this call, or possibly during this call. As well the new table (attached to an object as a its metatable) gets one additional reference, and so it won't be garbage collected during the call or just after it, as long as the object remains accessible to the Lua VM.
But there's a possible race condition if you happen to overwrite an existing metatable by itself: detaching the existing table could make it garbage collectable in the middle: the engine should make sure that the decrementation will not occur on the detached table and the incrementation will not occur on the attached table if they have identical pointers: setting a metatable to the current metatable should have no effect at all.