Re: Unexpected result using file:read("l")

Regarding the read-a-line at a time discussion. Some few years ago ago I wrote the small attached module to iterate through lines from a "text" file regardless of source, I have seen line endings such as \r\n from Windows, \n on Unix, and \r on Mac. The module allows you to iterate through all such files gracefully with io.lines. I have used this successfully on 10,000's of files from all sorts of places and systems.

This is inspired by David Manura's code here: http://lua-users.org/wiki/EnhancedFileLines, but takes it a couple of steps further.

It does NOT return the line ending at the end of the line to the calling function. I also treats the line ending on the last line in the file as optional.

Sorry if I am not allowed to attach a file to an email, I have been on this mailing list for ages, and this is the first time I felt I could contribute something meaningful.

I do have some code also for general handling of BOM's which I can supply if anyone is interested.

Regards

George Williams

On Mon, 28 Mar 2022 at 12:31, Scott Morgan <blumf@blueyonder.co.uk> wrote:

On 3/28/22 09:35, Thijs Schreijer wrote:
>
> I think the trend is that files become more and more cross-platform, and
> if that’s the case, then both options “*l” and “*L” are just becoming
> big foot guns because you cannot rely on them, and should either be
> removed from the implementation, or be made consistent across platforms.

Hummm... so Lua should automatically be able to detect if the file is in
EBCDIC encoding? (LF == 0x25)

How about handling UTF16 or UTF32 encodings? Should it automatically
skip the BOM mark on those files? (tip; Windows apps tend to drop in the
BOM on UTF8 encoded files, but not always)

At the least it should have support for the various local 8-bit code
pages, yeah? Convert them to your preferred encoding.

Where do you propose drawing the line?

Ultimately, there is no such thing as a "text file", just a messy
collection of conventions that may not hold, even within a single OS
install. It's down to you, the programmer, to understand this and
specify what is acceptable. It's not Lua's job.

Scott