Script speed improvement advice

Hi - first post here!

I could really do with some advice on how to speed up a script I've written.

Basically, the script does a string.find on a large piece of text (500 words-ish - sometimes more) for occurrences of any word from a list of over 40,000.

At the moment, I'm going through a table which contains the 40,000 "check words" and performing this string.find on each one.

Something similar to:

===========================

local #d = getTagValues(); --returns 40,000 entry table of string values

for i=1,#d do

m, document = checkForItem(d[i], document); --check to see if document contains word from table

if m == true then

if ll == "FULL" then

logln(os.date("%Y/%m/%d %H:%M:%S")..": Found item: "..d[i]);

end

===========================

This is taking anything up to 15 seconds or more. That sounds fairly quick, but this is actually a bottle neck in an extremely quick system.

Is there a quicker way to do what I'm doing here? Ideally it would do all 40k checks in a second - at the mo I would be grateful for 3 seconds!

I realise that if there isn't then I should perhaps consider a different methodology altogether - which I am doing, but we've invested quite a lot in this script and it would be great if it could really perform.

Thanks for any help.

Nathan Trevivian

___________________________________________________________________________

The information contained in this message is for the intended addressee onlyand may contain confidential and/or privileged information. If you are not the intended addressee, please delete this message and notify the sender; do not copy or distribute this message or disclose its contents to anyone.

Any views or opinions expressed in this message are those of the author and do not necessarily represent those of GateWest New Media Ltd.

____________________________________________________________________________