Re: avoiding copy - Luajit FFI

[Fabio Kaminski <fabiokaminski@...>]

On Tue, Jun 7, 2011 at 1:12 PM, Justin Cormack
<justin@specialbusservice.com> wrote:
> On Tue, Jun 7, 2011 at 4:58 PM, Fabio Kaminski <fabiokaminski@gmail.com>
> wrote:
>>
>> On Tue, Jun 7, 2011 at 12:53 PM, Justin Cormack
>> <justin@specialbusservice.com> wrote:
>> > On Tue, Jun 7, 2011 at 3:54 PM, Fabio Kaminski <fabiokaminski@gmail.com>
>> > wrote:
>> >>
>> >> Hi Folks,
>> >>
>> >> Im trying to get less memory copy as possible with luajit FFI, and in
>> >> the api to get your hands in a (void *) or (char *) pointer
>> >> is copying it through "ffi.string()" wich in turns, will create a lua
>> >> string..
>> >>
>> >> my first question: why cant ffi.string() reuse the given pointer.. vm
>> >> design? avoid concurrency races  from the C side?
>> >>
>> >> the c api "lua_pushlstring()" works the same way?
>> >>
>> >> second:
>> >>
>> >> When you simply need to push a string or even a file buffer thats
>> >> fine, but if whe are talking about things like byte buffers over a
>> >> network
>> >> those copy would make it loose a lot of performance in the long run..
>> >> even creating a bytebuffer on the C side and passing only the
>> >> requested byte string on the lua api side..
>> >>
>> >> Anyone has found a way (writing c logic or not) to avoid those copies?
>> >>
>> >
>> > Only call ffi.string when you absolutely need to, the rest of the time
>> > you
>> > can just pass around the raw buffers. You can just allocate them with
>> > ffi.new.
>> > Justin
>> >
>> >
>>
>> Yeah, thats the way to go.. create more C abstraction  (and black
>> boxes from lua app perspective) and try to switch C/Lua VM the least
>> as possible..
>>
>> no skinny C interface for now..
>>
>
> It is quite easy to accept either strings or buffers in your API, or have
> optional buffers, which will be used if they are provided. That way for
> performance critical parts the user can provide buffers, for other cases
> they just use strings. This also helps to avoid allocations in fast paths.
> Justin
>
looking from this perspective the ctypes from ffi do an excelente job..
like buffering mechanics, without have to write C lua api code for that ..
something not possible with immutable string design.. thats a win for sure..

but memory pointers are the digital currency of kernel/userspace
switches, or even third party apis, and my point of view is more
related
in that way..  get some constantly feeded buffer from C userland..

but looking in the ffi api we have this nice len argument in the
ffi.string function

str = ffi.string(ptr [,len])

So in theory you can break ptr in parts, and since you are allowed to
do pointer arithmetic in ptr
you can advance the cursor inside a lua context..

that's fast and can minimize the copies..