Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Robert Milkowski wrote On 06/28/06 15:52,: Hello Neil, Wednesday, June 21, 2006, 8:15:54 PM, you wrote: NP> Robert Milkowski wrote On 06/21/06 11:09,: Hello Neil, Why is this option available then? (Yes, that's a loaded question.) NP> I wouldn't call it an option, but an internal debug

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

So if you have a single thread doing open/write/close of 8K files and get 1.25MB/sec, that tells me you have something like a 6ms I/O latency. Which look reasonable also. What does iostat -x svc_t (client side) says ? 400ms seems high for the workload _and_ doesn't match my formula, so I don't li

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

To clarify what has just been stated. With zil disabled I got 4MB/sec. With zil enabled I get 1.25MB/sec On 6/23/06, Tao Chen <[EMAIL PROTECTED]> wrote: On 6/23/06, Roch <[EMAIL PROTECTED]> wrote: > > > > On Thu, Jun 22, 2006 at 04:22:22PM -0700, Joe Little wrote: > > > > On 6/22/06, Jeff Bonw

Re: Fwd: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On 6/23/06, Richard Elling <[EMAIL PROTECTED]> wrote: comment on analysis below...Tao Chen wrote:>   === Top 5 Devices with largest number of I/Os ===>>   DEVICE  READ AVG.ms MBWRITE AVG.ms MB  IOs SEEK >   ---  --- -- --  --- -- -

Re: Fwd: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

comment on analysis below... Tao Chen wrote: I should copy this to the list. -- Forwarded message -- On 6/23/06, * Joe Little* <[EMAIL PROTECTED] > wrote: I can post back to Roch what this latency is. I think the latency is a constant regard

Fwd: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

I should copy this to the list.-- Forwarded message --On 6/23/06, Joe Little <[EMAIL PROTECTED]> wrote: I can post back to Roch what this latency is. I think the latency is aconstant regardless of the zil or not. all that I do by disabling thezil is that I'm able to submit larger

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Joe Little wrote: On 6/23/06, Roch <[EMAIL PROTECTED]> wrote: Joe, you know this but for the benefit of others, I have to highlight that running any NFS server this way, may cause silent data corruption from client's point of view. Whenever a server keeps data in RAM this way and does not

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On 6/23/06, Roch <[EMAIL PROTECTED]> wrote: Joe Little writes: > On 6/22/06, Bill Moore <[EMAIL PROTECTED]> wrote: > > Hey Joe. We're working on some ZFS changes in this area, and if you > > could run an experiment for us, that would be great. Just do this: > > > > echo 'zil_disable/W

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On 6/23/06, Roch <[EMAIL PROTECTED]> wrote: Joe Little writes: > On 6/22/06, Bill Moore <[EMAIL PROTECTED]> wrote: > > Hey Joe. We're working on some ZFS changes in this area, and if you > > could run an experiment for us, that would be great. Just do this: > > > > echo 'zil_disable/W

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Joe Little writes: > On 6/22/06, Bill Moore <[EMAIL PROTECTED]> wrote: > > Hey Joe. We're working on some ZFS changes in this area, and if you > > could run an experiment for us, that would be great. Just do this: > > > > echo 'zil_disable/W1' | mdb -kw > > > > We're working on some f

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On 6/22/06, Bill Moore <[EMAIL PROTECTED]> wrote: Hey Joe. We're working on some ZFS changes in this area, and if you could run an experiment for us, that would be great. Just do this: echo 'zil_disable/W1' | mdb -kw We're working on some fixes to the ZIL so it won't be a bottleneck when

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, Jun 22, 2006 at 04:22:22PM -0700, Joe Little wrote: > Again, the issue is the multiple fsyncs that NFS requires, and likely > the serialization of those iscsi requests. Apparently, there is a > basic latency in iscsi that one could improve upon with FC, but we are > definitely in the all et

Re: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On 6/22/06, Jeff Bonwick <[EMAIL PROTECTED]> wrote: > a test against the same iscsi targets using linux and XFS and the > NFS server implementation there gave me 1.25MB/sec writes. I was about > to throw in the towel and deem ZFS/NFS has unusable until B41 came > along and at least gave me 1.25MB

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

> a test against the same iscsi targets using linux and XFS and the > NFS server implementation there gave me 1.25MB/sec writes. I was about > to throw in the towel and deem ZFS/NFS has unusable until B41 came > along and at least gave me 1.25MB/sec. That's still super slow -- is this over a 10Mb

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Yes, lockfs works. It uses the ZIL - unless it's disabled where it waits for all outstanding txgs to commit. The man page doesn't say it's specific to UFS, but does mention one specific UFS detail. Darren J Moffat wrote On 06/22/06 11:19,: Bill Sommerfeld wrote: On Thu, 2006-06-22 at 13:01, R

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, Jun 22, 2006 at 07:46:57PM +0200, Roch wrote: > > As I recall, the zfs sync is, unlike UFS, synchronous. Uh, are you talking about sync(2), or lockfs -f? IIRC, lockfs -f is always synchronous. Cheers, - jonathan -- Jonathan Adams, Solaris Kernel Development ___

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Yep. ZFS supports the ioctl (_FIOFFS) which 'lockfs -f' issues. -- Prabahar. Darren J Moffat wrote: > Bill Sommerfeld wrote: >> On Thu, 2006-06-22 at 13:01, Roch wrote: >>> Is there a sync command that targets individual FS ? >> >> Yes. lockfs -f > > Does lockfs work with ZFS ? The man page

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

As I recall, the zfs sync is, unlike UFS, synchronous. -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, Jun 22, 2006 at 06:19:20PM +0100, Darren J Moffat wrote: > Bill Sommerfeld wrote: > >On Thu, 2006-06-22 at 13:01, Roch wrote: > >> Is there a sync command that targets individual FS ? > > > >Yes. lockfs -f > > Does lockfs work with ZFS ? The man page appears to indicate it is very > UFS

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, 2006-06-22 at 13:19, Darren J Moffat wrote: > > Yes. lockfs -f > > Does lockfs work with ZFS ? The man page appears to indicate it is very > UFS specific. all of lockfs does not. but, if truss is to believed, the ioctl used by lockfs -f appears to. or at least, it returns without err

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Bill Sommerfeld wrote: On Thu, 2006-06-22 at 13:01, Roch wrote: Is there a sync command that targets individual FS ? Yes. lockfs -f Does lockfs work with ZFS ? The man page appears to indicate it is very UFS specific. -- Darren J Moffat ___ z

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, 2006-06-22 at 13:01, Roch wrote: > Is there a sync command that targets individual FS ? Yes. lockfs -f - Bill ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailm

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Bill Sommerfeld writes: > On Thu, 2006-06-22 at 03:55, Roch wrote: > > How about the 'deferred' option be on a leased basis with a > > deadline to revert to normal behavior; at most 24hrs at a > > time. > why? I'll trust your judgement over mine on this, so I won't press. But i

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Well, I should weigh in hear. I have been using ZFS with an iscsi backend and a NFS front end to my clients. Until B41 (not sure what fixed this) I was getting 20KB/sec for RAIDZ and 200KB/sec for just ZFS on on large iscsi LUNs (non-RAIDZ) when I was receiving many small writes, such as untarrin

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Thu, 2006-06-22 at 03:55, Roch wrote: > How about the 'deferred' option be on a leased basis with a > deadline to revert to normal behavior; at most 24hrs at a > time. why? > Console output everytime the option is enabled. in general, no. error messages to the console should be reserved

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Darren J Moffat wrote: > Bill Sommerfeld wrote: >> On Wed, 2006-06-21 at 14:15, Neil Perrin wrote: >>> Of course we would need to stress the dangers of setting 'deferred'. >>> What do you guys think? >> >> I can think of a use case for "deferred": improving the efficiency of a >> large mega-"transa

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Bill Sommerfeld wrote: On Wed, 2006-06-21 at 14:15, Neil Perrin wrote: Of course we would need to stress the dangers of setting 'deferred'. What do you guys think? I can think of a use case for "deferred": improving the efficiency of a large mega-"transaction"/batch job such as a nightly build

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

How about the 'deferred' option be on a leased basis with a deadline to revert to normal behavior; at most 24hrs at a time. Console output everytime the option is enabled. -r Torrey McMahon writes: > Neil Perrin wrote: > > > > Of course we would need to stress the dangers of setting 'd

RE: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Martin, Marcia R writes: > Did I miss something on this thread? Was the root cause of the > 15-minute fsync <> actually determined? > I think so ;-) -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Bill Sommerfeld wrote: On Wed, 2006-06-21 at 14:15, Neil Perrin wrote: Of course we would need to stress the dangers of setting 'deferred'. What do you guys think? I can think of a use case for "deferred": improving the efficiency of a large mega-"transaction"/batch job such as a nightly buil

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Neil, I think it might be wise to look at this problem from the perspective of an application (e.g. a simple database) designer taking into account all the new things that Solaris ZFS provides. In case of ZFS the designer does not have to worry about consistency of the on-disk file system format

RE: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

@opensolaris.org; Torrey McMahon Subject: Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved Neil Perrin wrote: > > > Robert Milkowski wrote On 06/21/06 11:09,: > >> Hello Neil, >> >>>> Why is this option available then? (Yes, that's a loaded question.) &

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Neil Perrin wrote: Robert Milkowski wrote On 06/21/06 11:09,: Hello Neil, Why is this option available then? (Yes, that's a loaded question.) NP> I wouldn't call it an option, but an internal debugging switch that I NP> originally added to allow progress when initially integrating the

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Neil Perrin wrote: Of course we would need to stress the dangers of setting 'deferred'. What do you guys think? That's the key: Be very explicit about what the option does and the side effects. ___ zfs-discuss mailing list zfs-discuss@opensolaris

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Wed, 2006-06-21 at 14:15, Neil Perrin wrote: > Of course we would need to stress the dangers of setting 'deferred'. > What do you guys think? I can think of a use case for "deferred": improving the efficiency of a large mega-"transaction"/batch job such as a nightly build. You create an initia

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Robert Milkowski wrote On 06/21/06 11:09,: Hello Neil, Why is this option available then? (Yes, that's a loaded question.) NP> I wouldn't call it an option, but an internal debugging switch that I NP> originally added to allow progress when initially integrating the ZIL. NP> As Roch says it

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Nicolas Williams wrote: On Wed, Jun 21, 2006 at 10:41:50AM -0600, Neil Perrin wrote: Why is this option available then? (Yes, that's a loaded question.) I wouldn't call it an option, but an internal debugging switch that I originally added to allow progress when initially integrating

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

On Wed, Jun 21, 2006 at 10:41:50AM -0600, Neil Perrin wrote: > >Why is this option available then? (Yes, that's a loaded question.) > > I wouldn't call it an option, but an internal debugging switch that I > originally added to allow progress when initially integrating the ZIL. > As Roch says it r

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Torrey McMahon wrote On 06/21/06 10:29,: Roch wrote: Sean Meighan writes: > The vi we were doing was a 2 line file. If you just vi a new file, add > one line and exit it would take 15 minutes in fdsynch. On recommendation > of a workaround we set > > set zfs:zil_disable=1 > > after

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Roch wrote: Sean Meighan writes: > The vi we were doing was a 2 line file. If you just vi a new file, add > one line and exit it would take 15 minutes in fdsynch. On recommendation > of a workaround we set > > set zfs:zil_disable=1 > > after the reboot the fdsynch is now < 0.1 seconds.

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Sean Meighan writes: > The vi we were doing was a 2 line file. If you just vi a new file, add > one line and exit it would take 15 minutes in fdsynch. On recommendation > of a workaround we set > > set zfs:zil_disable=1 > > after the reboot the fdsynch is now < 0.1 seconds. Now I have n

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

Well this does look more and more like a duplicate of: 6413510 zfs: writing to ZFS filesystem slows down fsync() on other files in the same FS Neil ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/

Re: [zfs-discuss] 15 minute fdsync problem and ZFS: Solved

The vi we were doing was a 2 line file. If you just vi a new file, add one line and exit it would take 15 minutes in fdsynch. On recommendation of a workaround we set set zfs:zil_disable=1 after the reboot the fdsynch is now < 0.1 seconds. Now I have no idea if it was this setting or the fact