RES: Full text search?

[André de Leiradella <leiradella@...>]

lua-bounces@bazar2.conectiva.com.br wrote:
> On Monday 28 February 2005 8:40 am, PA wrote:
>> Just as a follow-up, here is my first draft implementation of a
>> diminutive text search in Lua:
> 
> most people thought of something like this:
> 
> function addfile (ndx,fname)
>   local fl
>   local f = assert (io.open (fname))
>   for l in f:lines () do
>     for w in string.gfind (l, "%a+") do
>       ndx [w] = ndx [w] or {}
>       ndx [w] [fname] = ndx [w] [fname] + 1
>     end
>   end
>   f:close()
> end
> 
> after adding all the files, you can print how many times is a word in
> each file with:
> 
> for f,n in pairs (ndx["word"]) do
>   print (string.format ("%s times in file %s", n, f))
> end
> 
> but for big datasets, the index could get too big for memory.

Once more, maybe lper can help here. In lper, the heap is a memory
mapped file that grows when the application needs more heap. I know not
all the mapped file will fit in RAM if it's too big, but in an
application that uses too much memory the OS will start to use the swap,
won't it?

The benefit of lper is that when the application exits, the index is
left intact in the "heap" file, and will be ready to be used when the
app starts again, no need to load and parse config files or access
databases...

Andre