вівторок 28 березень 2006 05:27, Peter Jeremy написав:
> I'd suggest that each mmap be capable of storing several hundred msec of
> data as a minumum (maybe 10MB input and 5MB output, preferably more).
> Synchronisation can be done by writing tokens into pipes shared with the
> mmap's, optimised by
On Sat, 2006-Mar-25 21:39:27 +1100, Peter Jeremy wrote:
>What happens if you simulate read-ahead yourself? Have your main
>program fork and the child access pages slightly ahead of the parent
>but do nothing else.
I suspect something like this may be the best approach for your application.
My su
On Saturday 25 March 2006 06:46 pm, Peter Jeremy wrote:
= My guess is that the read-ahead algorithms are working but aren't doing
= enough re-ahead to cope with "read a bit, do some cpu-intensive processing
= and repeat" at 25MB/sec so you're winding up with a degree of serialisation
= where the I/
On Fri, 2006-Mar-24 15:18:00 -0500, Mikhail Teterin wrote:
>On the machine, where both mzip and the disk run at only 50%, the disk is a
>plain SATA drive (mzip's state goes from "RUN" to "vnread" and back).
...
> 18 usersLoad 0.46 0.53 0.60 24 ??? 15:15
>
>Mem:KBREAL
On Sat, 2006-Mar-25 09:20:13 -0500, Mikhail Teterin wrote:
>I'm sorry, that should be http://aldan.algebra.com/~mi/mzip.c -- I checked
>this time :-(
It doesn't look like it's doing anything especially weird. As Matt
pointed out, creating files with mmap() is not a good idea because the
syncer
On Sat, 2006-Mar-25 10:29:17 -0800, Matthew Dillon wrote:
>Really odd. Note that if your disk can only do 25 MBytes/sec, the
>calculation is: 2052167894 / 25MB = ~80 seconds, not ~60 seconds
>as you would expect from your numbers.
systat was reporting 25-26 MB/sec. dd'ing the underl
Mikhail Teterin wrote this message on Sat, Mar 25, 2006 at 09:20 -0500:
> = The downside is that touching an uncached page triggers a trap which may
> = not be as efficient as reading a block of data through the filesystem
> = interface, and I/O errors are delivered via signals (which may not be as
:The results here are weird. With 1GB RAM and a 2GB dataset, the
:timings seem to depend on the sequence of operations: reading is
:significantly faster, but only when the data was mmap'd previously
:There's one outlier that I can't easily explain.
:...
:Peter Jeremy
Really odd. Note that i
On Saturday 25 March 2006 05:39 am, Peter Jeremy wrote:
= On Fri, 2006-Mar-24 15:18:00 -0500, Mikhail Teterin wrote:
= >which there is not with the read. Read also requires fairly large
= >buffers in the user space to be efficient -- *in addition* to the
= >buffers in the kernel.
=
= I disagree.
On Fri, 2006-Mar-24 15:18:00 -0500, Mikhail Teterin wrote:
>which there is not with the read. Read also requires fairly large buffers in
>the user space to be efficient -- *in addition* to the buffers in the kernel.
I disagree. With a filesystem read, the kernel is solely responsible
for handli
On Fri, 2006-Mar-24 10:00:20 -0800, Matthew Dillon wrote:
>Ok. The next test is to NOT do umount/remount and then use a data set
>that is ~2x system memory (but can still be mmap'd by grep). Rerun
>the data set multiple times using grep and grep --mmap.
The results here are weird. W
> > May be the OS needs "reclaim-behind" for the sequential case?
> > This way you can mmap many many pages and use a much smaller
> > pool of physical pages to back them. The idea is for the VM
> > to reclaim pages N-k..N-1 when page N is accessed and allow
> > the same process to reuse this page
Matthew Dillon wrote:
> It is possible that the kernel believes the VM system to be too loaded
> to issue read-aheads, as a consequence of your blowing out of the system
> caches.
See attachment for the snapshot of `systat 1 -vm' -- it stays like that for
the most of the compression run tim
:On an amd64 system running about 6-week old -stable, both behave
:pretty much identically. In both cases, systat reports that the disk
:is about 96% busy whilst loading the cache. In the cache case, mmap
:is significantly faster.
:
:...
:turion% ls -l /6_i386/var/tmp/test
:-rw-r--r-- 1 peter
On Thu, 2006-Mar-23 15:16:11 -0800, Matthew Dillon wrote:
>FreeBSD. To determine which of the two is more likely, you have to
>run a smaller data set (like 600MB of data on a system with 1GB of ram),
>and use the unmount/mount trick to clear the cache before each grep test.
On an amd6
:I thought one serious advantage to this situation for sequential read
:mmap() is to madvise(MADV_DONTNEED) so that the pages don't have to
:wait for the clock hands to reap them. On a large Solaris box I used
:to have the non-pleasure of running the VM page scan rate was high, and
:I suggested t
> : time fgrep meowmeowmeow /home/oh.0.dump
> : 2.167u 7.739s 1:25.21 11.6% 70+3701k 23663+0io 6pf+0w
> : time fgrep --mmap meowmeowmeow /home/oh.0.dump
> : 1.552u 7.109s 2:46.03 5.2% 18+1031k 156+0io 106327pf+0w
> :
> :Use a big enough file to bust the memory caching (oh.
On Thu, Mar 23, 2006 at 03:16:11PM -0800, Matthew Dillon wrote:
> In anycase, this sort of test is not really a good poster child for how
> to use mmap(). Nobody in their right mind uses mmap() on datasets that
> they expect to be uncacheable and which are accessed sequentially. It's
:Yes, they both do work fine, but time gives very different stats for each. In
:my experiments, the total CPU time is noticably less with mmap, but the
:elapsed time is (much) greater. Here are results from FreeBSD-6.1/amd64 --
:notice the large number of page faults, because the system does no
четвер 23 березень 2006 15:48, Matthew Dillon Ви написали:
> Well, I don't know about FreeBSD, but both grep cases work just fine on
> DragonFly.
Yes, they both do work fine, but time gives very different stats for each. In
my experiments, the total CPU time is noticably less with mmap, b
:Actually, I can not agree here -- quite the opposite seems true. When running
:locally (no NFS involved) my compressor with the `-1' flag (fast, least
:effective compression), the program easily compresses faster, than it can
:read.
:
:The Opteron CPU is about 50% idle, *and so is the disk* pr
вівторок 21 березень 2006 17:48, Matthew Dillon Ви написали:
> Reading via mmap() is very well optimized.
Actually, I can not agree here -- quite the opposite seems true. When running
locally (no NFS involved) my compressor with the `-1' flag (fast, least
effective compression), the program
22 matches
Mail list logo
