I suppose I could be more specific and more helpful:
liolib.c needs to call pclose() instead of fclose() when the
file:close() metamethod is called. This is handled by aux_close:
http://www.lua.org/source/5.1/liolib.c.html#aux_close
The __close field is stored in the environment of each popen'ed
userdatum. This is done automatically because popen'ed userdata are
created only in io_popen() whose environment table is set explicitly
in luaopen_io().
Similarly, you could set an environment table on your function which
creates pipe userdata. Your get/set_timeout metamethods could use the
presense of this environment table (or a certain field contained in
it) to determine if a given userdata is indeed a pipe or something
else.
Note that if you want APR pipes to have the same semantics as
io.popen'ed pipes, then most of this work is already done for you ...
-Mark