Re: zero-copy read() into Lua string?

[Patrick Donnelly <batrick@...>]

On Sun, Jan 9, 2011 at 10:06 PM, Josh Haberman <jhaberman@gmail.com> wrote:
> There doesn't seem to be a way to construct a Lua string by writing
> directly into Lua's internal buffer.  For example, read_chars() in
> iolib uses the luaL_Buffer abstraction which uses an intermediate
> buffer.
>
> Is there any reason Lua couldn't support a function like:
>
>  /* Pushes a string of length "len" whose contents are uninitialized.
>   * The caller must initialize the string by writing to the returned
>   * buffer and call lua_finalizewstring() before calling any other Lua
>   * function on "L"; after this call, the client may no longer write
>   * into the returned buffer. */
>  char *lua_pushwstring(lua_State *L, size_t len);
>  void lua_finalizewstring(lua_State *L);
>
> (lua_finalizewstring() is there in case Lua wants to internally
> precompute a hash of the string or anything like that.)
>
> In some cases (like read()/fread()) this could save a memory copy for
> string data that is flowing from C->Lua, which seems like a big
> benefit.  Though I haven't implemented this idea to perform benchmarks,
> avoiding memory copies can improve efficiency significantly in my
> experience.

If you want to push a unknown string you either use luaL_Buffer or a
temporary userdata. Either way, you have to copy the string twice
(once into the buffer and again when Lua pushes the actual string).
There is no way around this. If you know the size of the string and it
is much greater than BUFSIZ, then you should use a temporary userdata
instead of luaL_Buffer (it is faster).

Note: In Lua 5.2, Lua uses temporary userdata for sizes greater than
BUFSIZ for the luaL_Buffer. This makes luaL_Buffers more attractive
for big (and small!) string building.

-- 
- Patrick Donnelly