Re: Luajit and curstom data structures

[KR <krunal.rao78@...>]

Mike Pall <mikelu-1104 <at> mike.de> writes:

> Ah, sorry. I forgot about a pending problem with empty proxy
> tables. The interpreter always creates proxied keys in the hash
> part, even if a metamethod is called (Lua has the same problem).
> The values are nil of course, which makes it indistinguishable
> from a truly empty hash table from the perspective of the user.
> But it made the hash table non-empty from the perspective of the
> JIT compiler.
> 
> I've fixed this problem in git HEAD. Thank you for the reminder!

Well, thank to you very much for implementing this optimization! 
I benchmarked the code again with the hit HEAD version and I can confirm that no
overhead is present anymore.

> 
> > -- I think this one is preferable 
> > local Vec = {}
> > function Vec.new(n) 
> >   local storage = ffi.new("double[?]", n) 
> >   local ret = {}
> >   -- {__index=storage, __newindex=storage} seems to works equally well...
> >   setmetatable(ret, 
> >   { __index = function(x, i) return storage[i] end
> >   , __newindex = function(x, i, v) storage[i]=v end })
> >   return ret
> > end
> > 
> > local Vec2 = {}
> > function Vec2.new(n) 
> >   local mt = {}
> >   mt.storage = ffi.new("double[?]", n)
> >   mt.__index = function(x, i) return mt.storage[i] end
> >   mt. __newindex = function(x, i, v) mt.storage[i]=v end
> >   local ret = {}
> >   setmetatable(ret, mt)
> >   return ret
> > end
> 
> These examples now run as fast as the direct cdata access.
> 
> But please note that you really don't want to create several new
> closures and tables for each instance. So this is not an optimal
> solution.

I am indeed trying to implement it with the metamethods for cdata approach
thanks to your recent work on the topic, but I am getting a bit confused about
the correct way to manage the memory managment (!!!) of these custom vectors and
vector views.

My first issue is that the following code:

local ffi = require("ffi")
local p = ffi.gc(ffi.C.malloc(10), ffi.C.free)

results in the following error (on WinXP, visual studio 2008, git HEAD built
with msvcbuild.bat): "missing declaration for symbol 'malloc'".

Moreover, in a previous post of yours I noticed you posted the following code:
return ffi.gc(ffi.new("void *[10]"), delarray) 

I thought that all values returned from ffi.new were always garbage collected,
does ffi.new being inside of a ffi.gc call change this behaviour?

Finally, at the moment I have the following (simplified) code:

local ffi = require("ffi")

ffi.cdef[[
typedef struct { 
  double* m_ptr;
  int32_t m_size;
} vec_t;
]]

local vec_mt = {
  __len = function(x) return x.m_size end,
  __index = function(x, i) return x.m_ptr[i] end,
  __newindex = function(x, i, v) x.m_ptr[i] = v end
}
local vec_ctor = ffi.metatype("vec_t", vec_mt)

-- Constructor for vector
function Vec(size)
  -- Should I use ffi.new here instead?
  local n = ffi.sizeof("double")*size
  local ptr = ffi.gc(ffi.C.malloc(n), ffi.C.free) 
  return vec_ctor(ptr, n)
end

Will the m_ptr inside of the vec_t keep the data of the vector alive (until the
value returned from Vec is alive) or will ffi.C.free be called at any time after
the execution of the Vec function is terminated?

> 
> > I am a bit more worried about the x[row][col] syntax, but maybe
> > it's doable as well if x[row] returns the "pointer" to the
> > element shifted row positions downward with respect to the
> > top-left element of the view which then gets shifted rightward
> > col elements (sorry for the imprecise language, I hope you got
> > the idea :P)
> 
> Actually the FFI library already uses exactly this approach for
> indexing multi-dimensional arrays. Ditto for arrays of structs or
> structs of arrays. The JIT compiler ought to be able to eliminate
> the intermediate pointer objects (this may be difficult in some
> use cases, though).
> 
> --Mike
> 
> 

That's excellent news! Afer I finish the vector and vector view objects I will
move to the corresponding matrix ones :)

Thank you for all your work and sorry for the (probably trivial) questions!



Kind regards,
KR