Re: How to calculate MD5 of huge files?

[Joshua Jensen <jjensen@...>]

----- Original Message -----
From: Luiz Henrique de Figueiredo
Date: 4/27/2010 5:05 AM
> > My version of lmd5 has this useful function.  It calculates the MD5 of
> > an entire file and does so from C code.
> >      
> >      const size_t BLOCK_SIZE = 32768;
> >      unsigned char* buffer;
> >      
> Use a fixed buffer of size BUFSIZ.  No need to malloc anything.
> fread buffers data anyway. No need to buffer it more than that.
>    
Well, this was written years ago, and it was tuned to certain data 
access patterns.

To check, according to Process Monitor running on Windows 7, I found 
that default fread() using BUFSIZ (512 bytes) reads in 4,096 bytes at a 
time.  fread(), then, hits the disk 8 times more often than the version 
of the code I have.  This is implementation dependent, of course, but 
there you go.
> >      fseek(file, 0, SEEK_END);
> >      
> Just read until fread returns zero. If you want to check for errors,
> call ferror at the end.
>    
I think the original code read the entire file into memory and 
calculated the MD5 in one fell swoop.  Then it evolved into its current 
form.

When I have some time to test the updates, I'll be sure to integrate the 
shorter forms you've suggested.

Thanks!

Josh