Re: ignoring BOM

[Robert Raschke <rtrlists@...>]

Hi Bulat,

I was hoping I could send a patch file, but unfortunately line endings got mangled at some point, so the whole file comes up as a diff, That's not really very useful. So here's just my change against Lua 5.1.2 for getting Lua to ignore a UTF-8 BOM.

All files get loaded via the function luaL_loadfile() in file lauxlib.c starting on line 486. I have just added a bit of code to deal with the BOM. It's not very pretty, but it works for me.

The whole function, including my addition now looks like this:


LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    lua_pushfstring(L, "@%s", filename);
    lf.f = fopen(filename, "r");
  }
  if (lf.f == NULL) return errfile(L, fnameindex);  /* unable to open file */
  c = ungetc(getc(lf.f), lf.f);
  if (!(isspace(c) || isprint(c)) && lf.f != stdin) {  /* binary file? */
    fclose(lf.f);
    lf.f = fopen(filename, "rb");  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, fnameindex); /* unable to reopen file */
  }

  /* vvv RTR vvv: Check for UTF-8 BOM ef bb bf */
  c = ungetc(getc(lf.f), lf.f);
  if (c == 0xef) {
    if (getc(lf.f) == 0xef && getc(lf.f) == 0xbb && getc(lf.f) == 0xbf) {
      /* do nothing, we've skipped the BOM and just continue with normal processing */
    } else {
     /* wasn't the UTF8 BOM, so reset everything again */
      fclose(lf.f);
      lf.f = fopen(filename, "rb");  /* reopen in binary mode */
      if (lf.f == NULL) return errfile(L, fnameindex); /* unable to reopen file */
    }
  }
  /* ^^^ RTR ^^^: Check for UTF-8 BOM ef bb bf */

  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (lf.f != stdin) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}


I hope this may be of use to you,
Robby