Re: io:lines() and \0

[Francisco Olarte <folarte@...>]

Hi:

On Mon, Feb 17, 2014 at 6:10 PM, Luiz Henrique de Figueiredo
<lhf@tecgraf.puc-rio.br> wrote:
>> I just noticed that io:lines() does not cope with \0 in the lines, and thus just returns truncated lines (lua-5.2.3, but legacy 5.1 likewise).
>>
>> May I suggest replacing the call to fgets in src/liolib.c so that we can read lines with \0 data?
>
> At least in Mac OS X and Linux fgets works just fine: it reads bytes
> until it sees \n, as promised in its man page. Unfortunately, fgets does
> not tell you how many bytes it has read and you're left with having to
> call strlen to find this out. I guess we could avoid strlen and use
> memchr instead.
>
> How did you propose to replace fgets?
>

I woud propose something like:

int c;
while ((c=getc(f))!='\n' && c!=EOF) {
    luaL_addchar(buffer, c);
}
// Test error if needed, we do not need strlen calls here, wrap buffer, return.

One of the strengths of C is the speed at what it can read a char and
act on it ( or not act ). getc() is normally very fast. lual_addchar
is very fast, for what I've seen in lualib.h.  The speed loss should
not be noticeable, and IMO not worrth the extra complexity in the
current readline function. I'f I've read the (5.2.2) sources correctly
the culprint is:

static int read_line (lua_State *L, FILE *f, int chop) {
  luaL_Buffer b;
  luaL_buffinit(L, &b);
  for (;;) {
    size_t l;
    char *p = luaL_prepbuffer(&b);
    if (fgets(p, LUAL_BUFFERSIZE, f) == NULL) {  /* eof? */
      luaL_pushresult(&b);  /* close buffer */
      return (lua_rawlen(L, -1) > 0);  /* check whether read something */
    }
    l = strlen(p);
    if (l == 0 || p[l-1] != '\n')
      luaL_addsize(&b, l);
    else {
      luaL_addsize(&b, l - chop);  /* chop 'eol' if needed */
      luaL_pushresult(&b);  /* close buffer */
      return 1;  /* read at least an `eol' */
    }
  }
}

Which, using getc, would reduce to,more or less:

static int read_line (lua_State *L, FILE *f, int chop) {
  luaL_Buffer b;
  luaL_buffinit(L, &b);
  int c;
  while (  (c=getc(f))!='\n'  &&  c!=EOF) {
    luaL_addchar(buffer, c);
  }
  if (!chop && c=='\n') {
    luaL_addchar(buffer, c); /* Add newline if needed. */
  }
  luaL_pushresult(&b);  /* close buffer */
  return (c=='\n') || (lua_rawlen(L, -1) > 0);  /* check whether read
something ( either newline or non-empty before EOF )*/
}

which I find much clearer.

Francisco Olarte.