On Jun 11, 2008, at 12:42 PM, Kirk Strauser wrote:
I'm almost ready to give up on this. I've gone as far as completely
rewriting the
original C++ program into straightforward C, and still the
performance is terrible on
FreeBSD versus Linux.
On Linux, GNU libc buffers file data much more extensively than
FreeBSD's libc does. It means that doing things like reading a dozen
bytes or so at a time is not intolerably slow on the former system,
but that doesn't mean that it's a great idea either.
If your data files are small enough to fit into 2GB of address space,
try using mmap() and then treat the file(s) as an array of records or
memoblocks or whatever, and let the VM system deal with paging in the
parts of the file you need. Otherwise, don't fread() 1 record at a
time, read in at least a (VM page / sizeof(record)) number of records
at a time into a bigger buffer, and then process that in RAM rather
than trying to fseek in little increments.
(This is the opposite of calling setvbuf() to set the I/O buffer to,
say, 13 bytes...)
Also, if you're malloc'ing and freeing buf & memohead with every
iteration of the loop, you're just thrashing the malloc system;
instead, allocate your buffers once before the loop, and reuse them
(zeroize or copy new data over the previous results) instead.
Regards,
--
-Chuck
Also note that on the FreeBSD machine, I have enough RAM that to
buffer the entire
file, and in practice gstat shows that the drives are idle for
subsequent runs after
the first one.
Right now my code looks a lot like:
for(recordnum = 0; recordnum < recordcount; recordnum++) {
buf = malloc(recordlength);
fread(buf, recordlength, 1, dbffile);
/* Do stuff with buf */
memoblock = getmemoblock(buf);
/* Skip to the requested block if we're not already there */
if(memoblock != currentmemofileblock) {
currentmemofileblock = memoblock;
fseek(memofile, currentmemofileblock * memoblocksize, SEEK_SET);
}
memohead = malloc(memoblocksize);
fread(memohead, memoblocksize, 1, memofile);
currentmemofileblock++;
/* Do stuff with memohead */
free(memohead);
free(buf);
}
...where recordlength == 13 in this one case. Given that the whole
file is buffered in
RAM, the small reads shouldn't make a difference, should they? I've
played with
setvbuf() and it shaves off a few percent of runtime, but nothing to
write home about.
Now, memofile gets quite a lot of seeks. Again, that shouldn't make
too much of a
difference if it's already buffered in RAM, should it? setvbuf() on
that file that
gets lots of random access actually made performance worse.
What else can I do to make my code run as well on FreeBSD as it does
on a much wimpier
Linux machine? I'm almost to the point of throwing in the towel and
making a Linux
server to do nothing more than run this one program if I can't
FreeBSD's performance
more on parity, and I honestly never thought I'd be considering that.
I'll gladly give shell access with my code and sample data files if
anyone is
interested in testing it.
--
Kirk Strauser
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]
"
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
