Re: Poor md5 module performance

[Michal Kolodziejczyk <miko@...>]

On 06.11.2010 21:04, Marcin Jurczuk wrote:

> The issue is that lua version is at least 10 times slower than python
> version. It is looks like this is somehow relate to how lua io:read
> reads file.

You are right, it looks like read function is throttled ("try to read
that much each time") to LUAL_BUFFERSIZE (which is BUFSIZ):

http://www.lua.org/source/5.1/liolib.c.html#read_chars

I think it is because:
- you rarely read in that big files in lua
- if you do, you want Lua to play nice with other applications in
mulithread OS.

So your program is slower not because Lua is slower (I hope it's not!),
but the lua code has to make 10M/8192=1280 system calls instead of just
one (as in python case).

> Is there some way to speed up this issue ??
> (I tried file:setvbuf option but it doesn't help at all)

You could:
1. change http://www.lua.org/source/5.1/luaconf.h.html#LUAL_BUFFERSIZE
to bigger value, ant then recomiple Lua (but I don't know if this could
have any unexpecvted side effect)
2. make a small C  module, which would load a file and return it as a
Lua string
3. Use a C module returning your file's MD5 sum

Regards,
miko