[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Using Lua to combine CSV files
- From: "chandan datta" <datta.chandan@...>
- Date: Mon, 25 Jun 2007 22:39:31 -0400
Hi Lua users,
I'm trying to read a group of CSV files in the following manner.I have list of huge log files( about 10-20 Mb stored as CSV) and the filenames are in a timestamped serial order inside a directory like
logfile1.csv,logfile2.csv,logfile3.csv ......... logfileN.csv .Inside the logfiles there is a field called TIMESTAMP and I want to combine the logfiles according to the timestamps.For example a TIMESTAMP(Linux timestamps are the number of seconds since midnight, January 1, 1970 GMT) value is 2 in
logfile2.csv and 15 in logfile15.csv. So I want to combine all the logfiles from logfile2.csv to logfile15.csv into one logfile.csv.
More detailed explanation:
I need to do some regular _expression_ matching before combining all the files(namely matching the timestamps).
So i need to develop a program which can work at the command prompt as :
$combinelogs timestamp1 timestamp2 logfile1.csv logfileN.csv outlogfile.csv
This would search the timestamp1 inside logfile1.csv and write all the records after a match into
outlogfile.csv and then write all the records from logfile2.csv ,logfile3.csv ........... till the timestamp2 inside logfileN.csv .
The Linux timestamps(e.g 1166212618.66 ) are the first fields in each record in the logfiles and to be provided by the user at the command prompt.
Please give some suggestions for an efficient way to do this since the log files are pretty large.I hope Lua can do this pretty efficiently.I'm on Suse Linux.