[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: Using Lua to combine CSV files
- From: "Shane Lee" <shaneplee@...>
- Date: Mon, 25 Jun 2007 22:00:59 -0500
On 6/25/07, chandan datta <email@example.com> wrote:
Hi Lua users,
I'm trying to read a group of CSV files in the following manner.I have list
of huge log files( about 10-20 Mb stored as CSV) and the filenames are in a
timestamped serial order inside a directory like
logfile1.csv,logfile2.csv,logfile3.csv ......... logfileN.csv .Inside the
logfiles there is a field called TIMESTAMP and I want to combine the
logfiles according to the timestamps.For example a TIMESTAMP(Linux
timestamps are the number of seconds since midnight, January 1, 1970 GMT)
value is 2 in logfile2.csv and 15 in logfile15.csv. So I want to combine all
the logfiles from logfile2.csv to logfile15.csv into one logfile.csv.
More detailed explanation:
I need to do some regular expression matching before combining all the
files(namely matching the timestamps).
So i need to develop a program which can work at the command prompt as :
$combinelogs timestamp1 timestamp2 logfile1.csv logfileN.csv outlogfile.csv
This would search the timestamp1 inside logfile1.csv and write all the
records after a match into outlogfile.csv and then write all the records
from logfile2.csv ,logfile3.csv ........... till the timestamp2 inside
The Linux timestamps(e.g 1166212618.66 ) are the first fields in each record
in the logfiles and to be provided by the user at the command prompt.
Please give some suggestions for an efficient way to do this since the log
files are pretty large.I hope Lua can do this pretty efficiently.I'm on Suse
I'm not really sure why you would send this to the Lua group. This is
something that could be handled in bash using grep and pipes. Granted
it would take a lot of testing and heck of a lot of planning, but I've
seen this sort of thing done before.
Am I missing something? Is this a project that MUST be done in Lua?