How big are the "chunks" that need to be inserted? If they are
relatively large, you can simply append each chunk to the end of the
file. But then keep a list of pointers to each chunk. The pointers
you can shuffle at will to be in the proper order, but each one
points to a different, nonsequential part of the data. You can then
keep the pointer list in a separate file, or in a dedicated large
space at the beginning of the data file.
David
I am still working on our new file i/o library and I have a
performance/reliability question. The specification of the file
format requires the data be sequential in the file... so if I want
to add data to a table at the beginning of the file I am required to
shift all of the following tables to make room for the new data.
Therefore, if a program was writing to two tables in the file in a
loop, there would be a lot of data shuffling... and it would get
worse the longer the loop ran. Does anyone have any experience with
this sort of thing? Am I worried about something that I wouldn't
even be able to notice in the end? Our files routinely reach
several hundred meg... and gigabyte files are not unheard of... what
would be the performance hit on trying to insert data at the
beginning of a several hundred meg file?
An alternative I am considering is to maintain each table in the
file in a separate temporary file while the file is open... then
when the file is closed LabVIEW would concatenate together the
temporary files to produce the output file according to the format
specification. The downside here is that I am now managing
who-knows-how-many temporary file refnums while the file is open...
this is not a huge deal from the library programming point of view,
but are there issues in labview with having many files open at the
same time? Are there other disadvantages with this approach I have
not thought of?
Finally, does anyone have some other suggestion for how I could
cleanly write the sorts of files I am describing here?
Thanks
George Gatling
Applied Technology Division, SFA Inc.
Space Physics Simulation Chamber
US Naval Research Laboratory
202-404-5405 (phone)
202-767-3553 (fax)
If trees could scream, would we be so cavalier about cutting them down?
We might, if they screamed all the time, for no good reason. --Jack Handy
--
David Ferster
Actimetrics, Inc.
1621 Elmwood Ave., Wilmette, IL 60091
http://www.actimetrics.com
847/922-2643 Phone
847/589-8103 FAX
