On 5/5/06, Eric Schrock <[EMAIL PROTECTED]> wrote:
On Fri, May 05, 2006 at 03:46:08PM -0700, Joe Little wrote: > Thanks for the tip. In the local case, I could send to the > iSCSI-backed ZFS RAIDZ at even faster rates, with a total elapsed time > of 50seconds (17 seconds better than UFS). However, I didn't even both > finishing the NFS client test, since it was taking a few seconds > between multiple 27K files. So, it didn't help NFS at all. I'm > wondering if there is something on the NFS end that needs changing, > no?Keep in mind that turning off this flag may corrupt on-disk state in the event of power loss, etc. What was the delta in the local case? 17 seconds better than UFS, but percentage wise how much faster than the original?
I believe it was only about 5-10% faster. I don't have the time results off hand, just some dtrace latency reports.
NFS has the property that it does an enormous amount of synchronous activity, which can tickle interesting pathologies. But it's strange that it didn't help NFS that much.
Should I also mount via async.. would this be honored on the Solaris
end? The other option mentioned with similar caveats was nocto. I just
tried with both, and the observed transfer rate was about 1.4k/s. It
was painful deleting the 3G directory via NFS, with about 100k/s
deletion rate on these 1000 files. Of course, When I went locally the
delete was instantaneous.
I'll give the complete results of a minute of this copy in dtrace results:
NFS3 op counts
==============
RFS3_SYMLINK 5
RFS3_MKDIR 27
RFS3_COMMIT 45
RFS3_RENAME 46
RFS3_CREATE 47
RFS3_ACCESS 126
RFS3_WRITE 143
RFS3_SETATTR 229
RFS3_GETATTR 536
RFS3_LOOKUP 1217
NFS3 op avg response time (usec)
================================
RFS3_LOOKUP 12
RFS3_ACCESS 13
RFS3_GETATTR 15
RFS3_WRITE 42
RFS3_RENAME 190052
RFS3_CREATE 199628
RFS3_COMMIT 244217
RFS3_SETATTR 256737
RFS3_MKDIR 258186
RFS3_SYMLINK 1150959
NFS3 op avg system time (usec)
==============================
RFS3_LOOKUP 10
RFS3_ACCESS 11
RFS3_GETATTR 12
RFS3_WRITE 40
RFS3_SETATTR 76
RFS3_COMMIT 100
RFS3_RENAME 101
RFS3_CREATE 104
RFS3_SYMLINK 105
RFS3_MKDIR 107
NFS3 op quantized response time (usec)
======================================
RFS3_ACCESS
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 106
16 |@@@@@@ 20
32 | 0
RFS3_WRITE
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@ 33
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 95
64 |@@@@ 15
128 | 0
RFS3_GETATTR
value ------------- Distribution ------------- count
2 | 0
4 | 1
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 347
16 |@@@@@@@@@@@@@ 176
32 |@ 11
64 | 0
128 | 1
256 | 0
RFS3_LOOKUP
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1060
16 |@@@@ 133
32 |@ 24
64 | 0
RFS3_SYMLINK
value ------------- Distribution ------------- count
32768 | 0
65536 |@@@@@@@@ 1
131072 |@@@@@@@@ 1
262144 | 0
524288 | 0
1048576 |@@@@@@@@ 1
2097152 |@@@@@@@@@@@@@@@@ 2
4194304 | 0
RFS3_MKDIR
value ------------- Distribution ------------- count
16384 | 0
32768 |@@@@ 3
65536 |@@@@@@@@@@@@@@@ 10
131072 |@@@@@@@@@@@@@@@ 10
262144 |@ 1
524288 | 0
1048576 |@@@@ 3
2097152 | 0
RFS3_CREATE
value ------------- Distribution ------------- count
256 | 0
512 |@@ 2
1024 | 0
2048 |@ 1
4096 | 0
8192 | 0
16384 |@ 1
32768 |@@@@@@@@@@@@@@@@@@@@ 23
65536 |@@@@@@@@@@@@ 14
131072 |@ 1
262144 | 0
524288 |@ 1
1048576 |@@@ 4
2097152 | 0
RFS3_RENAME
value ------------- Distribution ------------- count
256 | 0
512 |@ 1
1024 |@@ 2
2048 | 0
4096 | 0
8192 | 0
16384 | 0
32768 |@@@@@@@@@@@@@@@@@ 20
65536 |@@@@@@@@@@ 12
131072 |@@@@@ 6
262144 | 0
524288 |@ 1
1048576 |@@@ 4
2097152 | 0
RFS3_COMMIT
value ------------- Distribution ------------- count
256 | 0
512 |@ 1
1024 |@ 1
2048 | 0
4096 | 0
8192 | 0
16384 |@ 1
32768 |@@@@@@@@@@@@@@@@@ 19
65536 |@@@@@@@@@@@ 12
131072 |@@@@ 4
262144 | 0
524288 | 0
1048576 |@@@@@@ 7
2097152 | 0
RFS3_SETATTR
value ------------- Distribution ------------- count
256 | 0
512 |@ 4
1024 |@ 6
2048 | 0
4096 | 0
8192 | 0
16384 |@ 4
32768 |@@@@@@@@@@@@@@@@ 89
65536 |@@@@@@@@@ 53
131072 |@@@@@@ 34
262144 |@ 4
524288 |@ 3
1048576 |@@@@@ 30
2097152 | 1
4194304 | 0
NFS3 op quantized system time (usec)
====================================
RFS3_SYMLINK
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 5
128 | 0
RFS3_ACCESS
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 112
16 |@@@@ 14
32 | 0
RFS3_MKDIR
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 24
128 |@@@@ 3
256 | 0
RFS3_RENAME
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 41
128 |@@@@ 5
256 | 0
RFS3_COMMIT
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@ 18
64 |@@@@@@@@@ 10
128 |@@@@@@@@@@@@@@@ 17
256 | 0
RFS3_CREATE
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 41
128 |@@@@ 5
256 |@ 1
512 | 0
RFS3_WRITE
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@@@@@@ 50
32 |@@@@@@@@@@@@@@@@@@@@@@@ 84
64 |@@@ 9
128 | 0
RFS3_GETATTR
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@ 126
8 |@@@@@@@@@@@@@@@@@@@ 253
16 |@@@@@@@@@@@ 148
32 |@ 9
64 | 0
RFS3_LOOKUP
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1104
16 |@@@ 95
32 |@ 18
64 | 0
RFS3_SETATTR
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@ 81
64 |@@@@@@@@@@@@@@@@@@@@@@@@@ 143
128 |@ 4
256 | 0
> Also, how would one easily script the mdb command below to make > permanent? Unfortunately, there's no good way to do this, since the '::vdev' command isn't pipeable, and you'd have to use '::map' with hardcoded offsets that would be extremely brittle in the face of upgrades. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
