And of course, just to circle back, an rsync via ssh from the client to the Solaris ZFS/iscsi server came in at 17.5MB/sec, taking 1minute 16 seconds, or about 20% longer. So, NFS (over TCP) is 1.4k/s, and encrypted ssh is 17.5MB/sec following the same network path.
On 5/5/06, Joe Little <[EMAIL PROTECTED]> wrote:
On 5/5/06, Eric Schrock <[EMAIL PROTECTED]> wrote:
> On Fri, May 05, 2006 at 03:46:08PM -0700, Joe Little wrote:
> > Thanks for the tip. In the local case, I could send to the
> > iSCSI-backed ZFS RAIDZ at even faster rates, with a total elapsed time
> > of 50seconds (17 seconds better than UFS). However, I didn't even both
> > finishing the NFS client test, since it was taking a few seconds
> > between multiple 27K files. So, it didn't help NFS at all. I'm
> > wondering if there is something on the NFS end that needs changing,
> > no?
>
> Keep in mind that turning off this flag may corrupt on-disk state in the
> event of power loss, etc. What was the delta in the local case? 17
> seconds better than UFS, but percentage wise how much faster than the
> original?
>
I believe it was only about 5-10% faster. I don't have the time
results off hand, just some dtrace latency reports.
> NFS has the property that it does an enormous amount of synchronous
> activity, which can tickle interesting pathologies. But it's strange
> that it didn't help NFS that much.
Should I also mount via async.. would this be honored on the Solaris
end? The other option mentioned with similar caveats was nocto. I just
tried with both, and the observed transfer rate was about 1.4k/s. It
was painful deleting the 3G directory via NFS, with about 100k/s
deletion rate on these 1000 files. Of course, When I went locally the
delete was instantaneous.
I'll give the complete results of a minute of this copy in dtrace results:
NFS3 op counts
==============
RFS3_SYMLINK 5
RFS3_MKDIR 27
RFS3_COMMIT 45
RFS3_RENAME 46
RFS3_CREATE 47
RFS3_ACCESS 126
RFS3_WRITE 143
RFS3_SETATTR 229
RFS3_GETATTR 536
RFS3_LOOKUP 1217
NFS3 op avg response time (usec)
================================
RFS3_LOOKUP 12
RFS3_ACCESS 13
RFS3_GETATTR 15
RFS3_WRITE 42
RFS3_RENAME 190052
RFS3_CREATE 199628
RFS3_COMMIT 244217
RFS3_SETATTR 256737
RFS3_MKDIR 258186
RFS3_SYMLINK 1150959
NFS3 op avg system time (usec)
==============================
RFS3_LOOKUP 10
RFS3_ACCESS 11
RFS3_GETATTR 12
RFS3_WRITE 40
RFS3_SETATTR 76
RFS3_COMMIT 100
RFS3_RENAME 101
RFS3_CREATE 104
RFS3_SYMLINK 105
RFS3_MKDIR 107
NFS3 op quantized response time (usec)
======================================
RFS3_ACCESS
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 106
16 |@@@@@@ 20
32 | 0
RFS3_WRITE
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@ 33
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 95
64 |@@@@ 15
128 | 0
RFS3_GETATTR
value ------------- Distribution ------------- count
2 | 0
4 | 1
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@ 347
16 |@@@@@@@@@@@@@ 176
32 |@ 11
64 | 0
128 | 1
256 | 0
RFS3_LOOKUP
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1060
16 |@@@@ 133
32 |@ 24
64 | 0
RFS3_SYMLINK
value ------------- Distribution ------------- count
32768 | 0
65536 |@@@@@@@@ 1
131072 |@@@@@@@@ 1
262144 | 0
524288 | 0
1048576 |@@@@@@@@ 1
2097152 |@@@@@@@@@@@@@@@@ 2
4194304 | 0
RFS3_MKDIR
value ------------- Distribution ------------- count
16384 | 0
32768 |@@@@ 3
65536 |@@@@@@@@@@@@@@@ 10
131072 |@@@@@@@@@@@@@@@ 10
262144 |@ 1
524288 | 0
1048576 |@@@@ 3
2097152 | 0
RFS3_CREATE
value ------------- Distribution ------------- count
256 | 0
512 |@@ 2
1024 | 0
2048 |@ 1
4096 | 0
8192 | 0
16384 |@ 1
32768 |@@@@@@@@@@@@@@@@@@@@ 23
65536 |@@@@@@@@@@@@ 14
131072 |@ 1
262144 | 0
524288 |@ 1
1048576 |@@@ 4
2097152 | 0
RFS3_RENAME
value ------------- Distribution ------------- count
256 | 0
512 |@ 1
1024 |@@ 2
2048 | 0
4096 | 0
8192 | 0
16384 | 0
32768 |@@@@@@@@@@@@@@@@@ 20
65536 |@@@@@@@@@@ 12
131072 |@@@@@ 6
262144 | 0
524288 |@ 1
1048576 |@@@ 4
2097152 | 0
RFS3_COMMIT
value ------------- Distribution ------------- count
256 | 0
512 |@ 1
1024 |@ 1
2048 | 0
4096 | 0
8192 | 0
16384 |@ 1
32768 |@@@@@@@@@@@@@@@@@ 19
65536 |@@@@@@@@@@@ 12
131072 |@@@@ 4
262144 | 0
524288 | 0
1048576 |@@@@@@ 7
2097152 | 0
RFS3_SETATTR
value ------------- Distribution ------------- count
256 | 0
512 |@ 4
1024 |@ 6
2048 | 0
4096 | 0
8192 | 0
16384 |@ 4
32768 |@@@@@@@@@@@@@@@@ 89
65536 |@@@@@@@@@ 53
131072 |@@@@@@ 34
262144 |@ 4
524288 |@ 3
1048576 |@@@@@ 30
2097152 | 1
4194304 | 0
NFS3 op quantized system time (usec)
====================================
RFS3_SYMLINK
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 5
128 | 0
RFS3_ACCESS
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 112
16 |@@@@ 14
32 | 0
RFS3_MKDIR
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 24
128 |@@@@ 3
256 | 0
RFS3_RENAME
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 41
128 |@@@@ 5
256 | 0
RFS3_COMMIT
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@@@ 18
64 |@@@@@@@@@ 10
128 |@@@@@@@@@@@@@@@ 17
256 | 0
RFS3_CREATE
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 41
128 |@@@@ 5
256 |@ 1
512 | 0
RFS3_WRITE
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@@@@@@ 50
32 |@@@@@@@@@@@@@@@@@@@@@@@ 84
64 |@@@ 9
128 | 0
RFS3_GETATTR
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@ 126
8 |@@@@@@@@@@@@@@@@@@@ 253
16 |@@@@@@@@@@@ 148
32 |@ 9
64 | 0
RFS3_LOOKUP
value ------------- Distribution ------------- count
4 | 0
8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1104
16 |@@@ 95
32 |@ 18
64 | 0
RFS3_SETATTR
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@@@@@@@@@@ 81
64 |@@@@@@@@@@@@@@@@@@@@@@@@@ 143
128 |@ 4
256 | 0
>
> > Also, how would one easily script the mdb command below to make
> > permanent?
>
> Unfortunately, there's no good way to do this, since the '::vdev'
> command isn't pipeable, and you'd have to use '::map' with hardcoded
> offsets that would be extremely brittle in the face of upgrades.
>
> - Eric
>
> --
> Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
>
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
