ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC
instead of the Solaris page cache. But mmap() uses the latter. So if
anyone maps a file, ZFS has to keep the two caches in sync.
cp(1) uses mmap(2). When you use cp(1) it brings pages of the files it
copies into the Solaris
On Sat, Jul 4, 2009 at 06:03, Bob
Friesenhahn wrote:
> I am still trying to determine why Solaris 10 (Generic_141415-03) ZFS
> performs so terribly on my system. I blew a good bit of personal life
> savings on this set-up but am not seeing performance anywhere near what is
> expected. Testing wit
Mattias Pantzare wrote:
> > Performance when coping 236 GB of files (each file is 5537792 bytes, with
> > 20001 files per directory) from one directory to another:
> >
> > Copy Method Data Rate
> > ==
> > cpio -pd
After upgrading to b117 and doing a zfs upgrade -a the machine hang.
I pulled the power and booted it again.
Everything seems to work, except zfs userspace on a single filesystem.
Here is some info.
# uname -a
SunOS opensolaris 5.11 snv_117 i86pc i386 i86pc Solaris
# zfs upgrade
This system is
On Jul 4, 2009, at 12:03 AM, Bob Friesenhahn wrote:
% ./diskqual.sh
c1t0d0 130 MB/sec
c1t1d0 130 MB/sec
c2t202400A0B83A8A0Bd31 13422 MB/sec
c3t202500A0B83A8A0Bd31 13422 MB/sec
c4t600A0B80003A8A0B096A47B4559Ed0 191 MB/sec
c4t600A0B80003A8A0B096E47B456DAd0 192 MB/sec
c4t600A0B80003A8A0B00
On Jul 4, 2009, at 03:48, Phil Harman wrote:
The other thing that slows you down is that ZFS only flushes to disk
every 5 seconds if there are no synchronous writes. It would be
interesting to see iostat -xnz 1 while you are running your tests.
You may find the disks are writing very effici
Phil Harman wrote:
> ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC
> instead of the Solaris page cache. But mmap() uses the latter. So if
> anyone maps a file, ZFS has to keep the two caches in sync.
>
> cp(1) uses mmap(2). When you use cp(1) it brings pages of the files
On Sat, 4 Jul 2009, Jonathan Edwards wrote:
somehow i don't think that reading the first 64MB off (presumably) off a raw
disk device 3 times and picking the middle value is going to give you much
useful information on the overall state of the disks .. i believe this was
more of a quick hack t
On Sat, 4 Jul 2009, Phil Harman wrote:
ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC instead
of the Solaris page cache. But mmap() uses the latter. So if anyone maps a
file, ZFS has to keep the two caches in sync.
cp(1) uses mmap(2). When you use cp(1) it brings pages of
On Sat, 4 Jul 2009, Phil Harman wrote:
If you reboot, your cpio(1) tests will probably go fast again, until someone
uses mmap(2) on the files again. I think tar(1) uses read(2), but from my
iPod I can't be sure. It would be interesting to see how tar(1) performs if
you run that test before cp
On Sat, 4 Jul 2009, James Lever wrote:
Any insightful observations?
Probably multiple slog devices are used to expand slog size and not
used in parallel since that would require somehow knowing the order.
The principle bottleneck is likely the update rate of the first device
in the chain, f
On Sat, Jul 04, 2009 at 08:48:33AM +0100, Phil Harman wrote:
> ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC
> instead of the Solaris page cache. But mmap() uses the latter. So if
> anyone maps a file, ZFS has to keep the two caches in sync.
That's the first I've heard of
On Sat, Jul 4, 2009 at 1:38 AM, James Lever wrote:
>
> On 04/07/2009, at 1:49 PM, Ross Walker wrote:
>
>> I ran some benchmarks back when verifying this, but didn't keep them
>> unfortunately.
>>
>> You can google: XFS Barrier LVM OR EVMS and see the threads about this.
>
> Interesting reading. Te
A tar pipeline still provides terrible file copy performance. Read
bandwidth is only 26 MB. So I stopped the tar copy and re-tried the
cpio copy.
A second copy with the cpio results in a read/write data rate of only
54.9 MB/s (vs the just experienced 132 MB/s). Performance is reduced
by mo
Bob Friesenhahn wrote:
> A tar pipeline still provides terrible file copy performance. Read
> bandwidth is only 26 MB. So I stopped the tar copy and re-tried the
> cpio copy.
>
> A second copy with the cpio results in a read/write data rate of only
> 54.9 MB/s (vs the just experienced 132 MB
Ian Collins wrote:
Ross wrote:
Is that accounting for ZFS overhead? I thought it was more than that
(but of course, it's great news if not) :-)
A raidz2 pool with 8 500G drives showed 2.67GB free.
Same here. The ZFS overhead appears to be much smaller than similar UFS
filesystems.
E.g.
Joerg Schilling wrote:
Phil Harman wrote:
ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC
instead of the Solaris page cache. But mmap() uses the latter. So if
anyone maps a file, ZFS has to keep the two caches in sync.
cp(1) uses mmap(2). When you use cp(1) it bring
On Sat, 4 Jul 2009, Joerg Schilling wrote:
by more than half. Based on yesterday's experience, that may diminish
to only 33 MB/s.
"star -copy -no-fsync bs=8m fs=256m -C from-dir . to-dir"
is nearly 40% faster than
"find . | cpio -pdum to-dir"
Did you try to use highly perfor
Bob Friesenhahn wrote:
On Sat, 4 Jul 2009, Phil Harman wrote:
If you reboot, your cpio(1) tests will probably go fast again, until
someone uses mmap(2) on the files again. I think tar(1) uses read(2),
but from my iPod I can't be sure. It would be interesting to see how
tar(1) performs if you
Bob Friesenhahn wrote:
> On Sat, 4 Jul 2009, Joerg Schilling wrote:
> >> by more than half. Based on yesterday's experience, that may diminish
> >> to only 33 MB/s.
> >
> > "star -copy -no-fsync bs=8m fs=256m -C from-dir . to-dir"
> >
> > is nearly 40% faster than
> >
> > "find . | cpio
On Jul 4, 2009, at 11:57 AM, Bob Friesenhahn wrote:
This brings me to the absurd conclusion that the system must be
rebooted immediately prior to each use.
see Phil's later email .. an export/import of the pool or a remount of
the filesystem should clear the page cache - with mmap'd files
Gary Mills wrote:
On Sat, Jul 04, 2009 at 08:48:33AM +0100, Phil Harman wrote:
ZFS doesn't mix well with mmap(2). This is because ZFS uses the ARC
instead of the Solaris page cache. But mmap() uses the latter. So if
anyone maps a file, ZFS has to keep the two caches in sync.
That's
Ok, here is the scoop on the dire Solaris 10 (Generic_141415-03)
performance bug on my Sun Ultra 40-M2 attached to a StorageTek 2540
with latest firmware. I rebooted the system used cpio to send the
input files to /dev/null, and then immediately used cpio a second time
to send the input files
On Sat, 4 Jul 2009, Phil Harman wrote:
However, this is only part of the problem. The fundamental issue is that ZFS
has its own ARC apart from the Solaris page cache, so whenever mmap() is
used, all I/O to that file has to make sure that the two caches are in sync.
Hence, a read(2) on a file w
> "rw" == Ross Walker writes:
rw> Barriers are by default are disabled on ext3 mounts...
http://lwn.net/Articles/283161/
https://bugzilla.redhat.com/show_bug.cgi?id=458936
enabled by default on SLES. to enable on other distro:
mount -t ext3 -o barrier=1
(no help if using LVM2)
On Jul 4, 2009, at 14:30, Miles Nordin wrote:
yes, which is why it's worth suspecting knfsd as well. However I
don't think you can sell a Solaris system that performs 1/3 as well on
better hardware without a real test case showing the fast system's
broken.
It should be noted that RAID-0 perfo
Phil Harman wrote:
> I think Solaris (if you count SunOS 4.0, which was part of Solaris 1.0)
> was the first UNIX to get a working implementation of mmap(2) for files
> (if I recall correctly, BSD 4.3 had a manpage but no implementation for
> files). From that we got a whole lot of cool stuff,
Bob Friesenhahn wrote:
On Sat, 4 Jul 2009, Phil Harman wrote:
However, this is only part of the problem. The fundamental issue is
that ZFS has its own ARC apart from the Solaris page cache, so
whenever mmap() is used, all I/O to that file has to make sure that
the two caches are in sync. Hence
On Sat, 4 Jul 2009, Phil Harman wrote:
However, it seems that memory mapping is not responsible for the problem I
am seeing here. Memory mapping may make the problem seem worse, but it is
clearly not the cause.
mmap(2) is what brings ZFS files into the page cache. I think you've shown us
t
On Sat, 4 Jul 2009, Jonathan Edwards wrote:
this is only going to help if you've got problems in zfetch .. you'd probably
see this better by looking for high lock contention in zfetch with lockstat
This is what lockstat says when performance is poor:
Adaptive mutex spin: 477 events in 30.019
On Sat, 4 Jul 2009 13:03:52 -0500 (CDT)
Bob Friesenhahn wrote:
> On Sat, 4 Jul 2009, Joerg Schilling wrote:
> > Did you try to use highly performant software like star?
>
> No, because I don't want to tarnish your software's stellar
> reputation. I am focusing on Solaris 10 bugs today.
Blunt
Bob Friesenhahn wrote:
On Sat, 4 Jul 2009, Phil Harman wrote:
However, it seems that memory mapping is not responsible for the
problem I am seeing here. Memory mapping may make the problem seem
worse, but it is clearly not the cause.
mmap(2) is what brings ZFS files into the page cache. I
On Sat, 4 Jul 2009, Phil Harman wrote:
This is not a new problem. It seems that I have been banging my head
against this from the time I started using zfs.
I'd like to see mpstat 1 for each case, on an otherwise idle system,
but then there's probably a whole lot of dtrace I'd like to do ..
33 matches
Mail list logo
