Re: ignoring BOM

[Robert Raschke <rtrlists@...>]

Hi Bulat,

I was hoping I could send a patch file, but unfortunately line endings got mangled at some point, so the whole file comes up as a diff, That's not really very useful. So here's just my change against Lua 5.1.2 for getting Lua to ignore a UTF-8 BOM.

All files get loaded via the function luaL_loadfile() in file lauxlib.c starting on line 486. I have just added a bit of code to deal with the BOM. It's not very pretty, but it works for me.

The whole function, including my addition now looks like this:


LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
 LoadF lf;
 int status, readstatus;
 int c;
 int fnameindex = lua_gettop(L) + 1; /* index of filename on the stack */
 if (filename == NULL) {
ÂÂÂ lua_pushliteral(L, "=stdin");
ÂÂÂ lf.f = stdin;
 }
 else {
ÂÂÂ lua_pushfstring(L, "@%s", filename);
ÂÂÂ lf.f = fopen(filename, "r");
 }
 if (lf.f == NULL) return errfile(L, fnameindex); /* unable to open file */
 c = ungetc(getc(lf.f), lf.f);
 if (!(isspace(c) || isprint(c)) && lf.f != stdin) { /* binary file? */
ÂÂÂ fclose(lf.f);
ÂÂÂ lf.f = fopen(filename, "rb");Â /* reopen in binary mode */
ÂÂÂ if (lf.f == NULL) return errfile(L, fnameindex); /* unable to reopen file */
 }

 /* vvv RTR vvv: Check for UTF-8 BOM ef bb bf */
 c = ungetc(getc(lf.f), lf.f);
 if (c == 0xef) {
ÂÂÂ if (getc(lf.f) == 0xef && getc(lf.f) == 0xbb && getc(lf.f) == 0xbf) {
ÂÂÂÂÂ /* do nothing, we've skipped the BOM and just continue with normal processing */
ÂÂÂ } else {
ÂÂÂÂ /* wasn't the UTF8 BOM, so reset everything again */
ÂÂÂÂÂ fclose(lf.f);
ÂÂÂÂÂ lf.f = fopen(filename, "rb");Â /* reopen in binary mode */
ÂÂÂÂÂ if (lf.f == NULL) return errfile(L, fnameindex); /* unable to reopen file */
ÂÂÂ }
 }
 /* ^^^ RTR ^^^: Check for UTF-8 BOM ef bb bf */

 status = lua_load(L, getF, &lf, lua_tostring(L, -1));
 readstatus = ferror(lf.f);
 if (lf.f != stdin) fclose(lf.f); /* close file (even in case of errors) */
 if (readstatus) {
ÂÂÂ lua_settop(L, fnameindex);Â /* ignore results from `lua_load' */
ÂÂÂ return errfile(L, fnameindex);
 }
 lua_remove(L, fnameindex);
 return status;
}


I hope this may be of use to you,
Robby