RE: Data logging data base?

[Philippe Lhoste <PhiLho@...>]

John Passaniti wrote:
> > Because the data is sampled every three seconds for
> > several days, this data base could get large fast.
> > However I'm more interested in speed than disk space.
> > Tho I don't want have to take up all of the computer
> > memory with the data file.
> 
> Use the filesystem as the database.  That is:
> 
> > YEAR:MONTH:DAY:SECONDS_FROM_MIDNIGHT as the key, then
> > a channel number in array format, to get the values.
> 
> The directory structure might look like:
> 
> 	Year/
> 		Month/
> 			Day/
> 				Hour/
> 					Minute/
> 
> Within the final directory, either have each sample as a separate file, or
> bundle them into a single file.  I would probably make each a 
> separate file.

I am not a specialist of Unix, but where I work, we use a similar scheme
(sub-sub-directories, generated by hash, not by date) to store messages of users
for a web mail program.
It is nice and easy, but for very large databases, it seems there is a
problem of a limit of the number of available i-nodes.
Again, I don't know the exact figures, but you may be aware of this problem,
and perhaps aggregate data at a higher level (by hour or by day).

Of course, I assume here, as you do, that is on a Unix system. Windows or
other systems may have problems of their own.

[...]> So to get back the table of sample values for a particular time:
> 
> 	x = dofile("2001/7/19/2/6/9876543210")

Note: this works even on WinNT, ie. you can specify dofile("/2001/7/20/foo")
(current disk) or dofile("F:/2001/7/20/foo") (other disk) instead of
dofile([[F:\2001\7\20\foo]]). Great for compatiblity; and simplicity: the use of
backslashes forces to use the [[ ]] delimiters.
 
> > One of the other requirements is that I be able to
> > delete the data, on a day/date range bases.
> > Just put in NIL's?
> 
> Want to delete all data for July 19th, 2001?
> 
> 	rm -rf 2001/7/19

Again, assuming he is running on Unix... But it is easy to adapt to another
system.

---

lhf: Thank you for the pointer, I will have more precise knowledge now :-)

Regards.

-- 
--._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.--
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/
--´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`·._.·´¯`--

Sent through GMX FreeMail - http://www.gmx.net