Take Ignacio's advice. When crossing COM-apartments, either TypeLibrary (ie: IDispatch) marshalling occurs, or a custom COM-proxy is used. The only case where you could get violation of COM-based rules depends on the COM object you are calling, how it's implemented AND both the caller and callee are in an MTA (probably using the Free-Threaded Marshaller). Otherwise the values will get lost.
Even if your solution works on your machine, because you have all the source, no guarantees it will work on a "clean" machine.
I found another solution to my question.
I think this might be useful to other people who might encounter the same problem so I post this by replying here.
Although the use of CreatObject() described in section 6.2 of LuaCOM's user manual is luacom_obj = luacom.CreateObject(ID, creation_context, untyped), the use of untyped parameter seems never described in the manual.
After tracing LuaCOM's source code of CreateObject(), it seems that I can set untyped parameter as true to let LuaCOM treat the COM object to be created as a generic one without typeinfo and then I can get the value of EncodedData because all parameters are treated as [in, out].
It seems work in my case although I am not sure if this is really an alternative solution.
-- Jei-Wen Teng
2008/12/17 Ignacio Burgueño <email@example.com>
Jei-Wen Teng wrote:
I am trying to call a COM method from Lua by using LuaCOM.
The signature is as follow:
[out, retval] long* pVal);
Directional attribute [in], [out] is not specified.
EncodedData is the parameter that this method will store its result in.
That is wrong, see below.
My question is:
1. Will LuaCOM return all the values of VARIANT* type parameters?
No, because when no directional attribute is present, [in] is assumed 
2. If not, is there any way in LuaCOM to get the result value of EncodedData?
Unless you can change your method declaration, or add a new method with [in,out] parameters, you can't.
Also, you could create a new COM interface and wrap the calls to GetAPEncodedData
However, by not specifying [in,out] and changing EncodedData you're relying on not crossing apartments or worst, process boundaries. In the former, things may work under certain circumstances. In the latter, they won't work at all.