Re: [zfs-discuss] mpt errors on snv 127

[Chad Cantwell]

I don't think the hardware has any problems, it only started having errors when 
I upgraded OpenSolaris.
It's still working fine again now after a reboot.  Actually, I reread one of 
your earlier messages,
and I didn't realize at first when you said "non-Sun JBOD" that this didn't 
apply to me (in regards to
the msi=0 fix) because I didn't realize JBOD was shorthand for an external 
expander device.  Since
I'm just using baremetal, and passive backplanes, I think the msi=0 fix should 
apply to me based on
what you wrote earlier, anyway I've put 
set mpt:mpt_enable_msi = 0
now in /etc/system and rebooted as it was suggested earlier.  I've resumed my 
rsync, and so far there
have been no errors, but it's only been 20 minutes or so.  I should have a good 
idea by tomorrow if this
definitely fixed the problem (since even when the machine was not crashing it 
was tallying up iostat errors
fairly rapidly)

Thanks again for your help.  Sorry for wasting your time if the previously 
posted workaround fixes things.
I'll let you know tomorrow either way.

Chad

On Tue, Dec 01, 2009 at 05:57:28PM +1000, James C. McPherson wrote:
> Chad Cantwell wrote:
> >After another crash I checked the syslog and there were some different 
> >errors than the ones
> >I saw previously during operation:
> ...
> 
> >Nov 30 20:59:13 the-vault   LSI PCI device (1000,) not supported.
> ...
> >Nov 30 20:59:13 the-vault   mpt_config_space_init failed
> ...
> >Nov 30 20:59:15 the-vault   mpt_restart_ioc failed
> 
> 
> >Nov 30 21:33:02 the-vault fmd: [ID 377184 daemon.error] SUNW-MSG-ID: 
> >PCIEX-8000-8R, TYPE: Fault, VER: 1, SEVERITY: Major
> >Nov 30 21:33:02 the-vault EVENT-TIME: Mon Nov 30 21:33:02 PST 2009
> >Nov 30 21:33:02 the-vault PLATFORM: System-Product-Name, CSN: 
> >System-Serial-Number, HOSTNAME: the-vault
> >Nov 30 21:33:02 the-vault SOURCE: eft, REV: 1.16
> >Nov 30 21:33:02 the-vault EVENT-ID: 7886cc0d-4760-60b2-e06a-8158c3334f63
> >Nov 30 21:33:02 the-vault DESC: The transmitting device sent an invalid 
> >request.
> >Nov 30 21:33:02 the-vault   Refer to http://sun.com/msg/PCIEX-8000-8R for 
> >more information.
> >Nov 30 21:33:02 the-vault AUTO-RESPONSE: One or more device instances may be 
> >disabled
> >Nov 30 21:33:02 the-vault IMPACT: Loss of services provided by the device 
> >instances associated with this fault
> >Nov 30 21:33:02 the-vault REC-ACTION: Ensure that the latest drivers and 
> >patches are installed. Otherwise schedule a repair procedure to replace the 
> >affected device(s).  Us
> >e fmadm faulty to identify the devices or contact Sun for support.
> 
> 
> Sorry to have to tell you, but that HBA is dead. Or at
> least dying horribly. If you can't init the config space
> (that's the pci bus config space), then you've got about
> 1/2 the nails in the coffin hammered in. Then the failure
> to restart the IOC (io controller unit) == the rest of
> the lid hammered down.
> 
> 
> best regards,
> James C. McPherson
> --
> Senior Kernel Software Engineer, Solaris
> Sun Microsystems
> http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Chad Cantwell]

Well, ok, the msi=0 thing didn't help after all.  A few minutes after my last 
message a few errors showed
up in iostat, and then in a few minutes more the machine was locked up hard...  
Maybe I will try just
doing a scrub instead of my rsync process and see how that does.

Chad


On Tue, Dec 01, 2009 at 12:13:36AM -0800, Chad Cantwell wrote:
> I don't think the hardware has any problems, it only started having errors 
> when I upgraded OpenSolaris.
> It's still working fine again now after a reboot.  Actually, I reread one of 
> your earlier messages,
> and I didn't realize at first when you said "non-Sun JBOD" that this didn't 
> apply to me (in regards to
> the msi=0 fix) because I didn't realize JBOD was shorthand for an external 
> expander device.  Since
> I'm just using baremetal, and passive backplanes, I think the msi=0 fix 
> should apply to me based on
> what you wrote earlier, anyway I've put 
>   set mpt:mpt_enable_msi = 0
> now in /etc/system and rebooted as it was suggested earlier.  I've resumed my 
> rsync, and so far there
> have been no errors, but it's only been 20 minutes or so.  I should have a 
> good idea by tomorrow if this
> definitely fixed the problem (since even when the machine was not crashing it 
> was tallying up iostat errors
> fairly rapidly)
> 
> Thanks again for your help.  Sorry for wasting your time if the previously 
> posted workaround fixes things.
> I'll let you know tomorrow either way.
> 
> Chad
> 
> On Tue, Dec 01, 2009 at 05:57:28PM +1000, James C. McPherson wrote:
> > Chad Cantwell wrote:
> > >After another crash I checked the syslog and there were some different 
> > >errors than the ones
> > >I saw previously during operation:
> > ...
> > 
> > >Nov 30 20:59:13 the-vault   LSI PCI device (1000,) not supported.
> > ...
> > >Nov 30 20:59:13 the-vault   mpt_config_space_init failed
> > ...
> > >Nov 30 20:59:15 the-vault   mpt_restart_ioc failed
> > 
> > 
> > >Nov 30 21:33:02 the-vault fmd: [ID 377184 daemon.error] SUNW-MSG-ID: 
> > >PCIEX-8000-8R, TYPE: Fault, VER: 1, SEVERITY: Major
> > >Nov 30 21:33:02 the-vault EVENT-TIME: Mon Nov 30 21:33:02 PST 2009
> > >Nov 30 21:33:02 the-vault PLATFORM: System-Product-Name, CSN: 
> > >System-Serial-Number, HOSTNAME: the-vault
> > >Nov 30 21:33:02 the-vault SOURCE: eft, REV: 1.16
> > >Nov 30 21:33:02 the-vault EVENT-ID: 7886cc0d-4760-60b2-e06a-8158c3334f63
> > >Nov 30 21:33:02 the-vault DESC: The transmitting device sent an invalid 
> > >request.
> > >Nov 30 21:33:02 the-vault   Refer to http://sun.com/msg/PCIEX-8000-8R for 
> > >more information.
> > >Nov 30 21:33:02 the-vault AUTO-RESPONSE: One or more device instances may 
> > >be disabled
> > >Nov 30 21:33:02 the-vault IMPACT: Loss of services provided by the device 
> > >instances associated with this fault
> > >Nov 30 21:33:02 the-vault REC-ACTION: Ensure that the latest drivers and 
> > >patches are installed. Otherwise schedule a repair procedure to replace 
> > >the affected device(s).  Us
> > >e fmadm faulty to identify the devices or contact Sun for support.
> > 
> > 
> > Sorry to have to tell you, but that HBA is dead. Or at
> > least dying horribly. If you can't init the config space
> > (that's the pci bus config space), then you've got about
> > 1/2 the nails in the coffin hammered in. Then the failure
> > to restart the IOC (io controller unit) == the rest of
> > the lid hammered down.
> > 
> > 
> > best regards,
> > James C. McPherson
> > --
> > Senior Kernel Software Engineer, Solaris
> > Sun Microsystems
> > http://blogs.sun.com/jmcp   http://www.jmcp.homeunix.com/blog
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Mark Nipper]

This is basically just a me too.  I'm using different hardware but essentially 
the same problems.  The relevant hardware I have is:
---
SuperMicro MBD-H8Di3+-F-O motherboard with LSI 1068E onboard
SuperMicro SC846E2-R900B 4U chassis with two LSI SASx36 expander chips on the 
backplane
24 Western Digital RE4-GP 2TB 7.2k RPM SATA drives
---

I have two SFF-8087 to SFF-8087 cables running from the two ports on the 
motherboard (4 channels each) to two ports on the backplane, each port going to 
one of the LSI expander chips.  The backplane has four additional ports which 
support cascading additional enclosures together, but I'm not making use of any 
of this at the moment.

The machine is currently dead at the data center, and it's late, so if you want 
anything more from me, just let me know and I'll run stuff tomorrow on the 
machine.  But otherwise, the behavior sounds the same as all of the other MPT 
reports recently.  I was not seeing these types of problems with 2009.06, but 
also wanted to upgrade to get raidz3 support.

Just tell me what other commands you might want output from to help diagnose 
the problem.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Is write(2) made durable atomically?

[Robert Milkowski]

Neil Perrin wrote:




Under the hood in ZFS, writes are committed using either shadow 
paging or

logging, as I understand it. So I believe that I mean to ask whether a
write(2), pushed to ZPL, and pushed on down the stack, can be split 
into
multiple transactions? Or, instead, is it guaranteed to be committed 
in a

single transaction, and so committed atomically?


A write made through the ZPL (zfs_write()) will be broken into 
transactions
that contain at most 128KB user data. So a large write could well be 
split

across transaction groups, and thus committed separately.


So what happens if application is doing a synchronous write of lets say 
512KB?
The write will be splitted in at least 4 separate transactions and the 
write will be confirmed to the application only after all 512KB has been 
written. But is there a possibility that if after a first transaction 
was commited system crashed and although write was not confirmed to the 
application 128KB of it has been commited to the disk? Or will it be 
rolled back? Basically for synchronous writes of more than 128KB - is it 
guaranteed that all data under a given write is committed or nothing at all?


--
Robert Milkowski
http://milek.blogspot.com


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] ZFS dedup clarification

[Dominic Kay]

You have fallen into the same trap I fell into.  df(1M) is not dedup aware;
dedup occurs at the pool level, not the filesystem level. If you look at
your df output, you can see your disk seems to be growing in size which is
non-intuitive.

 Once you start using ZFS and in particular dedup but also other data
services such as snapshots, you need to start using the zpool and zfs
reporting commands and abandon df(1M).
/d


2009/11/27 Chavdar Ivanov 

> 2009/11/27 Thomas Maier-Komor :
> > Chavdar Ivanov schrieb:
> >> Hi,
> >>
> >> I BFUd successfully snv_128 over snv_125:
> >>
> >> ---
> >> # cat /etc/release
> >>   Solaris Express Community Edition snv_125 X86
> >>Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
> >> Use is subject to license terms.
> >> Assembled 05 October 2009
> >> # uname -a
> >> SunOS cheeky 5.11 snv_128 i86pc i386 i86pc
> >> ...
> >>
> >> being impatient to test zfs dedup. I was able to set dedup=on (I presume
> with the default sha256 key) on a few filesystems and did the following
> trivial test (this is an edited script session):
> >>
> >> Script started on Wed Oct 28 09:38:38 2009
> >> # zfs get dedup rpool/export/home
> >> NAME   PROPERTY  VALUE SOURCE
> >> rpool/export/home  dedup onlocal
> >> # for i in 1 2 3 4 5 ; do mkdir /export/home/d${i} && df -k
> /export/home/d${i} && zfs get used rpool/export/home && cp /testfile
> /export/home/d${i}; done
> >> Filesystemkbytesused   avail capacity  Mounted on
> >> rpool/export/home17418240  27 6063425 1%/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  used  27K-
> >> Filesystemkbytesused   avail capacity  Mounted on
> >> rpool/export/home17515512  103523 6057381 2%/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  used  102M   -
> >> Filesystemkbytesused   avail capacity  Mounted on
> >> rpool/export/home17682840  271077 6056843 5%/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  used  268M   -
> >> Filesystemkbytesused   avail capacity  Mounted on
> >> rpool/export/home17852184  442345 6054919 7%/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  used  432M   -
> >> Filesystemkbytesused   avail capacity  Mounted on
> >> rpool/export/home17996580  587996 6053933 9%/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  used  574M   -
> >> # zfs get all rpool/export/home
> >> NAME   PROPERTY  VALUE  SOURCE
> >> rpool/export/home  type  filesystem -
> >> rpool/export/home  creation  Mon Sep 21  9:27 2009  -
> >> rpool/export/home  used  731M   -
> >> rpool/export/home  available 5.77G  -
> >> rpool/export/home  referenced731M   -
> >> rpool/export/home  compressratio 1.00x  -
> >> rpool/export/home  mounted   yes-
> >> rpool/export/home  quota none   default
> >> rpool/export/home  reservation   none   default
> >> rpool/export/home  recordsize128K   default
> >> rpool/export/home  mountpoint/export/home
> inherited from rpool/export
> >> rpool/export/home  sharenfs  offdefault
> >> rpool/export/home  checksum  on default
> >> rpool/export/home  compression   offdefault
> >> rpool/export/home  atime on default
> >> rpool/export/home  devices   on default
> >> rpool/export/home  exec  on default
> >> rpool/export/home  setuidon default
> >> rpool/export/home  readonly  offdefault
> >> rpool/export/home  zoned offdefault
> >> rpool/export/home  snapdir   hidden default
> >> rpool/export/home  aclmode   groupmask  default
> >> rpool/export/home  aclinheritrestricted default
> >> rpool/export/home  canmount  on default
> >> rpool/export/home  shareiscsioffdefault
> >> rpool/export/home  xattr on default
> >> rpool/export/home  copies1  default
> >> rpool/export/home  version   4  -
> >> rpool/export/home  utf8only  off 

Re: [zfs-discuss] mpt errors on snv 127

[Mark Johnson]

Chad Cantwell wrote:

Hi,

 I was using for quite awhile OpenSolaris 2009.06
with the opensolaris-provided mpt driver to operate a zfs raidz2 pool of
about ~20T and this worked perfectly fine (no issues or device errors
logged for several months, no hanging).  A few days ago I decided to
reinstall with the latest OpenSolaris in order to take advantage of
raidz3.  


Just to be clear... The same setup was working fine on osol2009.06,
you upgraded to b127 and it started failing?

Did you keep the osol2009.06 be around so you can reboot back to it?

If so, have you tried the osol2009.06 mpt driver in the
BE with the latest bits (make sure you make a backup copy
of the mpt driver)?



MRJ


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Mark Johnson]

Mark Johnson wrote:



Chad Cantwell wrote:

Hi,

 I was using for quite awhile OpenSolaris 2009.06
with the opensolaris-provided mpt driver to operate a zfs raidz2 pool of
about ~20T and this worked perfectly fine (no issues or device errors
logged for several months, no hanging).  A few days ago I decided to
reinstall with the latest OpenSolaris in order to take advantage of
raidz3.  


Just to be clear... The same setup was working fine on osol2009.06,
you upgraded to b127 and it started failing?

Did you keep the osol2009.06 be around so you can reboot back to it?

If so, have you tried the osol2009.06 mpt driver in the
BE with the latest bits (make sure you make a backup copy
of the mpt driver)?


What's the earliest build someone has seen this
problem? i.e. if we binary chop, has anyone seen it in
b118?

I have no idea if the old mpt drivers will work on a
new kernel... But if someone wants to try... Something
like the following should work...


# first, I would work out of a test BE in case you
# mess something up.
beadm create test-be
beadm activate test-be
reboot

# assuming your lasted BE is call snv127, mount it and backup
# the stock mpt driver and conf file.
beadm mount snv127 /mnt
cp /mnt/kernel/drv/mpt.conf /mnt/kernel/drv/mpt.conf.orig
cp /mnt/kernel/drv/amd64/mpt /mnt/kernel/drv/amd64/mpt.orig

# see what builds are out there...
pkg search /kernel/drv/amd64/mpt


# There's probably an easier way to do this...
# grab an older mpt. This will take a while since it's
# not in it's own package and ckr has some dependencies
# so it will pull in a bunch of other packages.
# change out 118 with the build you want to grab.
mkdir /tmp/mpt
pkg image-create -f -F -a opensolaris.org=http://pkg.opensolaris.org/dev 
/tmp/mpt
pkg -R /tmp/mpt/ install sunw...@0.5.11-0.118
cp /tmp/mpt/kernel/drv/mpt.conf /mnt/kernel/drv/mpt.conf
cp /tmp/mpt/kernel/drv/amd64/mpt /mnt/kernel/drv/amd64/mpt
rm -rf /tmp/mpt/
bootadm update-archive -R /mnt




MRJ



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Markus Kovero]

We actually tried this, although using sol10-version of mpt-driver. 
Surprisingly it didn't work :-)

Yours
Markus Kovero

-Original Message-
From: zfs-discuss-boun...@opensolaris.org 
[mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Mark Johnson
Sent: 1. joulukuuta 2009 15:57
To: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] mpt errors on snv 127



Mark Johnson wrote:
> 
> 
> Chad Cantwell wrote:
>> Hi,
>>
>>  I was using for quite awhile OpenSolaris 2009.06
>> with the opensolaris-provided mpt driver to operate a zfs raidz2 pool of
>> about ~20T and this worked perfectly fine (no issues or device errors
>> logged for several months, no hanging).  A few days ago I decided to
>> reinstall with the latest OpenSolaris in order to take advantage of
>> raidz3.  
> 
> Just to be clear... The same setup was working fine on osol2009.06,
> you upgraded to b127 and it started failing?
> 
> Did you keep the osol2009.06 be around so you can reboot back to it?
> 
> If so, have you tried the osol2009.06 mpt driver in the
> BE with the latest bits (make sure you make a backup copy
> of the mpt driver)?

What's the earliest build someone has seen this
problem? i.e. if we binary chop, has anyone seen it in
b118?

I have no idea if the old mpt drivers will work on a
new kernel... But if someone wants to try... Something
like the following should work...


# first, I would work out of a test BE in case you
# mess something up.
beadm create test-be
beadm activate test-be
reboot

# assuming your lasted BE is call snv127, mount it and backup
# the stock mpt driver and conf file.
beadm mount snv127 /mnt
cp /mnt/kernel/drv/mpt.conf /mnt/kernel/drv/mpt.conf.orig
cp /mnt/kernel/drv/amd64/mpt /mnt/kernel/drv/amd64/mpt.orig

# see what builds are out there...
pkg search /kernel/drv/amd64/mpt


# There's probably an easier way to do this...
# grab an older mpt. This will take a while since it's
# not in it's own package and ckr has some dependencies
# so it will pull in a bunch of other packages.
# change out 118 with the build you want to grab.
mkdir /tmp/mpt
pkg image-create -f -F -a opensolaris.org=http://pkg.opensolaris.org/dev 
/tmp/mpt
pkg -R /tmp/mpt/ install sunw...@0.5.11-0.118
cp /tmp/mpt/kernel/drv/mpt.conf /mnt/kernel/drv/mpt.conf
cp /tmp/mpt/kernel/drv/amd64/mpt /mnt/kernel/drv/amd64/mpt
rm -rf /tmp/mpt/
bootadm update-archive -R /mnt




MRJ



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Adam Cheal]

> 
> What's the earliest build someone has seen this
> problem? i.e. if we binary chop, has anyone seen it
> in
> b118?
> 

We have used every "stable" build from b118 up, as b118 was the first reliable 
one that could be used is a CIFS-heavy environment. The problem occurs on all 
of them.

- Adam
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Travis Tabbal]

If someone from Sun will confirm that it should work to use the mpt driver from 
2009.06, I'd be willing to set up a BE and try it. I still have the snapshot 
from my 2009.06 install, so I should be able to mount that and grab the files 
easily enough.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Workaround for mpt timeouts in snv_127

[Travis Tabbal]

Just an update, my scrub completed without any timeout errors in the log. XVM 
with MSI disabled globally.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Workaround for mpt timeouts in snv_127

[Travis Tabbal]

Perhaps. As I noted though, it also occurs on the onboard NVidia SATA 
controller when MSI is enabled. I had already put a line in /etc/system to 
disable MSI for that controller per a forum thread and it worked great. I'm now 
running with all MSI disabled via XVM as the mpt controller is giving me the 
same problems. As it's happening on totally different controller types, cable 
types, and drive types, I have to go with software issues. I know for sure the 
NVidia issue didn't come up on 2009.06. It makes the system take forever to 
boot, so it's very noticeable. It happened when I first went to dev builds, I 
want to say it was around b118. I updated for better XVM support for newer 
Linux kernels. 

The NVidia controller causes similar log messages. Command timeouts. Disabling 
MSIs fixes it as well. Motherboard is an Asus M4N82 Deluxe. NVIDIA nForce 980a 
SLI chipset.

I expect the root cause is the same, and I would guess that something is 
causing the drivers to miss or not receive some interrupts. However, my 
programming at this level is limited, so perhaps I'm misdiagnosing the issue.
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] How many TLDs should you have?

[Christopher White]

All,

We're going to start testing ZFS and I had a question about Top Level
Devices (TLDs).  In Sun's class, they specifically said not to use more than
9 TLDs due to performance concerns.  Our storage admins make LUNs roughly
15G in size -- so how would we make a large pool (1TB) if we're limited to
only 9 TLDs?

The Best Practices guide (
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations)
suggests not using 40+ disks in a RAIDZ TLD:

"Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration with
one logical device of 40+ devices. See the sections below for examples of
redundant configurations."

Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each RAIDZ?

Or is the 9 TLDs an old recommendation that has since been changed?

Thanks!

Chris
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Adding drives to system - disk labels not consistent

[Stuart Reid]

Hi -

Using OpenSolaris 2008.11

Hope this is enough information.

Stuart


-Original Message-
From: cindy.swearin...@sun.com [mailto:cindy.swearin...@sun.com]
Sent: November 30, 2009 11:32 AM
To: Stuart Reid
Cc: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] Adding drives to system - disk labels not
consistent


Hi Stuart,

Which Solaris release are you seeing this behavior?

I would like reproduce it and file a bug, if necessary.

Thanks,

Cindy

On 11/29/09 13:06, Stuart Reid wrote:
> Answered by own question...
>
> When using the -n switch the output is truncated i.e. the d0 is not
printed. When actually adding the drives, the operation is as expected -

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] CR# 6574286, remove slog device

[Moshe Vainer]

Thanks Pablo. I think I confused the matters - i meant to respond to the issue 
in bug #6733267, and somehow landed on that one...

-Original Message-
From: Pablo Méndez Hernández [mailto:pabl...@gmail.com]
Sent: Monday, November 30, 2009 12:35 PM
To: Moshe Vainer
Cc: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] CR# 6574286, remove slog device

Hi Moshe:

On Mon, Nov 30, 2009 at 20:30, Moshe Vainer  wrote:
> Any news on this bug? We are trying to implement write acceleration, but 
> can't deploy to production with this issue still not fixed. If anyone has an 
> estimate (e.g., would it be part of 10.02?) i would very much appreciate to 
> know.

According to:

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6574286

the bug was fixed in snv_125, so it will be in 10.03.


Cheers.

--

Pablo Méndez Hernández
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] CR# 6574286, remove slog device

[Moshe Vainer]

George, thank you very much! This is great news.

-Original Message-
From: george.wil...@sun.com [mailto:george.wil...@sun.com]
Sent: Monday, November 30, 2009 9:04 PM
To: Moshe Vainer
Cc: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] CR# 6574286, remove slog device

Moshe Vainer wrote:
> I am sorry, i think i confused the matters a bit. I meant the bug that 
> prevents importing with slog device missing, 6733267.
> I am aware that one can remove a slog device, but if you lose your rpool and 
> the device goes missing while you rebuild, you will lose your pool in its 
> entirety. Not a situation to be tolerated in production.

Expect the fix for this issue this month.

Thanks,
George
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Cindy Swearingen]

Hi Chris,

If you have 40 or so disks then you would create 5-6 RAIDZ virtual
devices of 7-8 disks each, or possibly include two disks for the root
pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks
for a non-root pool.

This configuration guide hasn't been updated for RAIDZ-3 yet, but you
will get some ideas about how to configure a redundant configuration
of many disks, here:

http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide

See ZFS Configuration Example (x4500 with raidz2)

Cindy

On 12/01/09 09:20, Christopher White wrote:

All,

We're going to start testing ZFS and I had a question about Top Level 
Devices (TLDs).  In Sun's class, they specifically said not to use more 
than 9 TLDs due to performance concerns.  Our storage admins make LUNs 
roughly 15G in size -- so how would we make a large pool (1TB) if we're 
limited to only 9 TLDs?


The Best Practices guide ( 
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations 
) suggests not using 40+ disks in a RAIDZ TLD:


"Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration 
with one logical device of 40+ devices. See the sections below for 
examples of redundant configurations."


Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each RAIDZ?

Or is the 9 TLDs an old recommendation that has since been changed?

Thanks!

Chris




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Christopher White]

Cindy --

Thanks for the link!

I see in one of the examples that there are 14 TLDs (all mirrored).  Does
that mean there are no performance issues with having more than 9 TLDs?  In
the Sun class I attended, the instructor said to not use more than 9 TLDs,
which seems like it could be very limiting, especially in a SAN setting.
Like I said, our storage group presents 15G LUNs to use -- so it'd be
difficult to keep the TLDs under 9 and have a very large filesystem.

Let me know what you think.  Thanks!

Chris



On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen
wrote:

> Hi Chris,
>
> If you have 40 or so disks then you would create 5-6 RAIDZ virtual
> devices of 7-8 disks each, or possibly include two disks for the root
> pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks
> for a non-root pool.
>
> This configuration guide hasn't been updated for RAIDZ-3 yet, but you
> will get some ideas about how to configure a redundant configuration
> of many disks, here:
>
> http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide
>
> See ZFS Configuration Example (x4500 with raidz2)
>
> Cindy
>
>
> On 12/01/09 09:20, Christopher White wrote:
>
>> All,
>>
>> We're going to start testing ZFS and I had a question about Top Level
>> Devices (TLDs).  In Sun's class, they specifically said not to use more than
>> 9 TLDs due to performance concerns.  Our storage admins make LUNs roughly
>> 15G in size -- so how would we make a large pool (1TB) if we're limited to
>> only 9 TLDs?
>>
>> The Best Practices guide (
>> http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations)
>>  suggests not using 40+ disks in a RAIDZ TLD:
>>
>> "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration
>> with one logical device of 40+ devices. See the sections below for examples
>> of redundant configurations."
>>
>> Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each
>> RAIDZ?
>>
>> Or is the 9 TLDs an old recommendation that has since been changed?
>>
>> Thanks!
>>
>> Chris
>>
>>
>> 
>>
>> ___
>> zfs-discuss mailing list
>> zfs-discuss@opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
>
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Richard Elling]

On Dec 1, 2009, at 8:53 AM, Christopher White wrote:


Cindy --

Thanks for the link!

I see in one of the examples that there are 14 TLDs (all mirrored).   
Does that mean there are no performance issues with having more than  
9 TLDs?  In the Sun class I attended, the instructor said to not use  
more than 9 TLDs, which seems like it could be very limiting,  
especially in a SAN setting.  Like I said, our storage group  
presents 15G LUNs to use -- so it'd be difficult to keep the TLDs  
under 9 and have a very large filesystem.


Let me know what you think.  Thanks!


I've never heard of a top-level vdev limit.  Are you sure they are not  
getting

confused with the leaf-vdev suggestions in the zpool(1m) man page?
 -- richard



Chris



On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen > wrote:

Hi Chris,

If you have 40 or so disks then you would create 5-6 RAIDZ virtual
devices of 7-8 disks each, or possibly include two disks for the root
pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks)  
disks

for a non-root pool.

This configuration guide hasn't been updated for RAIDZ-3 yet, but you
will get some ideas about how to configure a redundant configuration
of many disks, here:

http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide

See ZFS Configuration Example (x4500 with raidz2)

Cindy


On 12/01/09 09:20, Christopher White wrote:
All,

We're going to start testing ZFS and I had a question about Top  
Level Devices (TLDs).  In Sun's class, they specifically said not to  
use more than 9 TLDs due to performance concerns.  Our storage  
admins make LUNs roughly 15G in size -- so how would we make a large  
pool (1TB) if we're limited to only 9 TLDs?


The Best Practices guide ( http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations 
 ) suggests not using 40+ disks in a RAIDZ TLD:


"Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored  
configuration with one logical device of 40+ devices. See the  
sections below for examples of redundant configurations."


Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each  
RAIDZ?


Or is the 9 TLDs an old recommendation that has since been changed?

Thanks!

Chris




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Cindy Swearingen]

Chris,

The TLD terminology is confusing so let's think about this way:

Performance is best when only 3-9 physical devices are included in
each mirror or RAIDZ grouping as shown in the configuration guide.

Cindy

On 12/01/09 09:53, Christopher White wrote:

Cindy --

Thanks for the link!

I see in one of the examples that there are 14 TLDs (all mirrored).  
Does that mean there are no performance issues with having more than 9 
TLDs?  In the Sun class I attended, the instructor said to not use more 
than 9 TLDs, which seems like it could be very limiting, especially in a 
SAN setting.  Like I said, our storage group presents 15G LUNs to use -- 
so it'd be difficult to keep the TLDs under 9 and have a very large 
filesystem.


Let me know what you think.  Thanks!

Chris



On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen 
mailto:cindy.swearin...@sun.com>> wrote:


Hi Chris,

If you have 40 or so disks then you would create 5-6 RAIDZ virtual
devices of 7-8 disks each, or possibly include two disks for the root
pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks
for a non-root pool.

This configuration guide hasn't been updated for RAIDZ-3 yet, but you
will get some ideas about how to configure a redundant configuration
of many disks, here:

http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide

See ZFS Configuration Example (x4500 with raidz2)

Cindy


On 12/01/09 09:20, Christopher White wrote:

All,

We're going to start testing ZFS and I had a question about Top
Level Devices (TLDs).  In Sun's class, they specifically said
not to use more than 9 TLDs due to performance concerns.  Our
storage admins make LUNs roughly 15G in size -- so how would we
make a large pool (1TB) if we're limited to only 9 TLDs?

The Best Practices guide (

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations
) suggests not using 40+ disks in a RAIDZ TLD:

"Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored
configuration with one logical device of 40+ devices. See the
sections below for examples of redundant configurations."

Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in
each RAIDZ?

Or is the 9 TLDs an old recommendation that has since been changed?

Thanks!

Chris




___
zfs-discuss mailing list
zfs-discuss@opensolaris.org 
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Carson Gaspar]

Travis Tabbal wrote:

If someone from Sun will confirm that it should work to use the mpt
driver from 2009.06, I'd be willing to set up a BE and try it. I
still have the snapshot from my 2009.06 install, so I should be able
to mount that and grab the files easily enough.


I tried, it doesn't work. It's interesting to note that the itmpt driver 
(much, much older) works just fine. It seems someone has gotten 
"creative" with the mpt driver's use of the DDI.


--
Carson
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Bob Friesenhahn]

On Tue, 1 Dec 2009, Cindy Swearingen wrote:


The TLD terminology is confusing so let's think about this way:

Performance is best when only 3-9 physical devices are included in
each mirror or RAIDZ grouping as shown in the configuration guide.


It seems that these "TLDs" are what the rest of us call "vdev"s.

Performance improves with more "vdev"s since zfs load-shares across 
them.  The load-sharing helps defend against performance loss due to 
poorly-performing devices.


Each write to a vdev produces one I/O operation to each of the devices 
in the vdev.  While adding more devices to raidz1/raidz2 vdevs 
increases storage capacity, the increase is at a loss to the number of 
IOPS which can be sustained.  Variability in device performance allows 
a poorly-performing device to dominate the performance of the whole 
vdev, and particulary for raidz1/raidz2, which need to be written to 
synchronously.  The loss of IOPS, and the risk of performance loss due 
to imperfectly-matched hardware, results in increased risk of 
performance loss with too many devices in a raidz1/raidz2 vdev.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Christopher White]

Bob - thanks, that makes sense.

The classroom book refers to "top-level virtual devices," and were referred
to as TLDs throughout the class (Top-Level Devices).  As you noted, those
are either the base LUN, mirror, raidz, or raidz2.

So there's no limit to the number of TLDs/vdevs we can have, the only
recommendation is that we have no more than 3-9 LUNs per RAIDZ vdev?

Chris

On Tue, Dec 1, 2009 at 12:25 PM, Bob Friesenhahn <
bfrie...@simple.dallas.tx.us> wrote:

> On Tue, 1 Dec 2009, Cindy Swearingen wrote:
>
>  The TLD terminology is confusing so let's think about this way:
>>
>> Performance is best when only 3-9 physical devices are included in
>> each mirror or RAIDZ grouping as shown in the configuration guide.
>>
>
> It seems that these "TLDs" are what the rest of us call "vdev"s.
>
> Performance improves with more "vdev"s since zfs load-shares across them.
>  The load-sharing helps defend against performance loss due to
> poorly-performing devices.
>
> Each write to a vdev produces one I/O operation to each of the devices in
> the vdev.  While adding more devices to raidz1/raidz2 vdevs increases
> storage capacity, the increase is at a loss to the number of IOPS which can
> be sustained.  Variability in device performance allows a poorly-performing
> device to dominate the performance of the whole vdev, and particulary for
> raidz1/raidz2, which need to be written to synchronously.  The loss of IOPS,
> and the risk of performance loss due to imperfectly-matched hardware,
> results in increased risk of performance loss with too many devices in a
> raidz1/raidz2 vdev.
>
> Bob
> --
> Bob Friesenhahn
> bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
>
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Adding drives to system - disk labels not consistent

[Cindy Swearingen]

I was able to reproduce this problem on the latest Nevada build:


# zpool create tank raidz c1t2d0 c1t3d0 c1t4d0
# zpool add -n tank raidz c1t5d0 c1t6d0 c1t7d0
would update 'tank' to the following configuration:
tank
  raidz1
c1t2d0
c1t3d0
c1t4d0
  raidz1
c1t5
c1t6
c1t7

I will file a low-priority bug since it is a -n display problem
only.

Thanks for reporting it.

Cindy


On 11/30/09 18:15, Stuart Reid wrote:

Hi -

Using OpenSolaris 2008.11

Hope this is enough information.

Stuart


-Original Message-
From: cindy.swearin...@sun.com [mailto:cindy.swearin...@sun.com]
Sent: November 30, 2009 11:32 AM
To: Stuart Reid
Cc: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] Adding drives to system - disk labels not
consistent


Hi Stuart,

Which Solaris release are you seeing this behavior?

I would like reproduce it and file a bug, if necessary.

Thanks,

Cindy

On 11/29/09 13:06, Stuart Reid wrote:

Answered by own question...

When using the -n switch the output is truncated i.e. the d0 is not

printed. When actually adding the drives, the operation is as expected -


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Bob Friesenhahn]

On Tue, 1 Dec 2009, Christopher White wrote:


So there's no limit to the number of TLDs/vdevs we can have, the only 
recommendation is
that we have no more than 3-9 LUNs per RAIDZ vdev?


Yes.  No one here has complained about problems due to too many vdevs. 
They have complained due to too many pools on one system, too many 
filesystems in a pool, and too many disks in one raidz2 vdef.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Chad Cantwell]

First I tried just upgrading to b127, that had a few issues besides the mpt 
driver.  After that
I did a clean install of b127, but no I don't have my osol2009.06 root still 
there.  I wasn't
sure how to install another copy and leave it there (I suspect it is possible, 
since I saw
when doing upgrades it creates a second root environment, but my forte isn't 
solaris so I
just reformatted the root device)

On Tue, Dec 01, 2009 at 08:09:32AM -0500, Mark Johnson wrote:
> 
> 
> Chad Cantwell wrote:
> >Hi,
> >
> > I was using for quite awhile OpenSolaris 2009.06
> >with the opensolaris-provided mpt driver to operate a zfs raidz2 pool of
> >about ~20T and this worked perfectly fine (no issues or device errors
> >logged for several months, no hanging).  A few days ago I decided to
> >reinstall with the latest OpenSolaris in order to take advantage of
> >raidz3.
> 
> Just to be clear... The same setup was working fine on osol2009.06,
> you upgraded to b127 and it started failing?
> 
> Did you keep the osol2009.06 be around so you can reboot back to it?
> 
> If so, have you tried the osol2009.06 mpt driver in the
> BE with the latest bits (make sure you make a backup copy
> of the mpt driver)?
> 
> 
> 
> MRJ
> 
> 
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] mpt errors on snv 127

[Chad Cantwell]

To update everyone, I did a complete zfs scrub, and it it generated no errors 
in iostat, and I have 4.8T of
data on the filesystem so it was a fairly lengthy test.  The machine also has 
exhibited no evidence of
instability.  If I were to start copying a lot of data to the filesystem again 
though, I'm sure it would
generate errors and crash again.

Chad


On Tue, Dec 01, 2009 at 12:29:16AM -0800, Chad Cantwell wrote:
> Well, ok, the msi=0 thing didn't help after all.  A few minutes after my last 
> message a few errors showed
> up in iostat, and then in a few minutes more the machine was locked up 
> hard...  Maybe I will try just
> doing a scrub instead of my rsync process and see how that does.
> 
> Chad
> 
> 
> On Tue, Dec 01, 2009 at 12:13:36AM -0800, Chad Cantwell wrote:
> > I don't think the hardware has any problems, it only started having errors 
> > when I upgraded OpenSolaris.
> > It's still working fine again now after a reboot.  Actually, I reread one 
> > of your earlier messages,
> > and I didn't realize at first when you said "non-Sun JBOD" that this didn't 
> > apply to me (in regards to
> > the msi=0 fix) because I didn't realize JBOD was shorthand for an external 
> > expander device.  Since
> > I'm just using baremetal, and passive backplanes, I think the msi=0 fix 
> > should apply to me based on
> > what you wrote earlier, anyway I've put 
> > set mpt:mpt_enable_msi = 0
> > now in /etc/system and rebooted as it was suggested earlier.  I've resumed 
> > my rsync, and so far there
> > have been no errors, but it's only been 20 minutes or so.  I should have a 
> > good idea by tomorrow if this
> > definitely fixed the problem (since even when the machine was not crashing 
> > it was tallying up iostat errors
> > fairly rapidly)
> > 
> > Thanks again for your help.  Sorry for wasting your time if the previously 
> > posted workaround fixes things.
> > I'll let you know tomorrow either way.
> > 
> > Chad
> > 
> > On Tue, Dec 01, 2009 at 05:57:28PM +1000, James C. McPherson wrote:
> > > Chad Cantwell wrote:
> > > >After another crash I checked the syslog and there were some different 
> > > >errors than the ones
> > > >I saw previously during operation:
> > > ...
> > > 
> > > >Nov 30 20:59:13 the-vault   LSI PCI device (1000,) not supported.
> > > ...
> > > >Nov 30 20:59:13 the-vault   mpt_config_space_init failed
> > > ...
> > > >Nov 30 20:59:15 the-vault   mpt_restart_ioc failed
> > > 
> > > 
> > > >Nov 30 21:33:02 the-vault fmd: [ID 377184 daemon.error] SUNW-MSG-ID: 
> > > >PCIEX-8000-8R, TYPE: Fault, VER: 1, SEVERITY: Major
> > > >Nov 30 21:33:02 the-vault EVENT-TIME: Mon Nov 30 21:33:02 PST 2009
> > > >Nov 30 21:33:02 the-vault PLATFORM: System-Product-Name, CSN: 
> > > >System-Serial-Number, HOSTNAME: the-vault
> > > >Nov 30 21:33:02 the-vault SOURCE: eft, REV: 1.16
> > > >Nov 30 21:33:02 the-vault EVENT-ID: 7886cc0d-4760-60b2-e06a-8158c3334f63
> > > >Nov 30 21:33:02 the-vault DESC: The transmitting device sent an invalid 
> > > >request.
> > > >Nov 30 21:33:02 the-vault   Refer to http://sun.com/msg/PCIEX-8000-8R 
> > > >for more information.
> > > >Nov 30 21:33:02 the-vault AUTO-RESPONSE: One or more device instances 
> > > >may be disabled
> > > >Nov 30 21:33:02 the-vault IMPACT: Loss of services provided by the 
> > > >device instances associated with this fault
> > > >Nov 30 21:33:02 the-vault REC-ACTION: Ensure that the latest drivers and 
> > > >patches are installed. Otherwise schedule a repair procedure to replace 
> > > >the affected device(s).  Us
> > > >e fmadm faulty to identify the devices or contact Sun for support.
> > > 
> > > 
> > > Sorry to have to tell you, but that HBA is dead. Or at
> > > least dying horribly. If you can't init the config space
> > > (that's the pci bus config space), then you've got about
> > > 1/2 the nails in the coffin hammered in. Then the failure
> > > to restart the IOC (io controller unit) == the rest of
> > > the lid hammered down.
> > > 
> > > 
> > > best regards,
> > > James C. McPherson
> > > --
> > > Senior Kernel Software Engineer, Solaris
> > > Sun Microsystems
> > > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
> > ___
> > zfs-discuss mailing list
> > zfs-discuss@opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[zfs-discuss] zpool import - device names not always updated?

[Ragnar Sundblad]

It seems that device names aren't always updated when importing
pools if devices have moved. I am not sure if this is only an
cosmetic issue or if it could actually be a real problem -
could it lead to the device not being found at a later import?

/ragge

(This is on snv_127.)

I ran the following script:

#!/bin/bash

set -e
set -x

zfs create -V 1G rpool/vol1
zfs create -V 1G rpool/vol2
zpool create pool mirror /dev/zvol/dsk/rpool/vol1 /dev/zvol/dsk/rpool/vol2
zpool status pool
zpool export pool
zfs create rpool/subvol1
zfs create rpool/subvol2
zfs rename rpool/vol1 rpool/subvol1/vol1
zfs rename rpool/vol2 rpool/subvol2/vol2
zpool import -d /dev/zvol/dsk/rpool/subvol1
sleep 1
zpool import -d /dev/zvol/dsk/rpool/subvol2
sleep 1
zpool import -d /dev/zvol/dsk/rpool/subvol1 pool
zpool status pool


And got the output below. I have annotated it with ### remarks.


# bash zfs-test.bash
+ zfs create -V 1G rpool/vol1
+ zfs create -V 1G rpool/vol2
+ zpool create pool mirror /dev/zvol/dsk/rpool/vol1 /dev/zvol/dsk/rpool/vol2
+ zpool status pool
  pool: pool
 state: ONLINE
 scrub: none requested
config:

NAME  STATE READ WRITE CKSUM
pool  ONLINE   0 0 0
  mirror-0ONLINE   0 0 0
/dev/zvol/dsk/rpool/vol1  ONLINE   0 0 0
/dev/zvol/dsk/rpool/vol2  ONLINE   0 0 0

errors: No known data errors
+ zpool export pool
+ zfs create rpool/subvol1
+ zfs create rpool/subvol2
+ zfs rename rpool/vol1 rpool/subvol1/vol1
+ zfs rename rpool/vol2 rpool/subvol2/vol2
+ zpool import -d /dev/zvol/dsk/rpool/subvol1
  pool: pool
id: 13941781561414544058
 state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices.  The
fault tolerance of the pool may be compromised if imported.
   see: http://www.sun.com/msg/ZFS-8000-2Q
config:

pool  DEGRADED
  mirror-0DEGRADED
/dev/zvol/dsk/rpool/subvol1/vol1  ONLINE
/dev/zvol/dsk/rpool/vol2  UNAVAIL  cannot open
### Note that it can't find vol2 - which is expected.
+ sleep 1
### The sleep here seems to be necessary for vol1 to magically be
### found in the next zpool import.
+ zpool import -d /dev/zvol/dsk/rpool/subvol2
  pool: pool
id: 13941781561414544058
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

pool  ONLINE
  mirror-0ONLINE
/dev/zvol/dsk/rpool/vol1  ONLINE
/dev/zvol/dsk/rpool/subvol2/vol2  ONLINE
### Note that it says vol1 is ONLINE, under it's old path, though it actually 
has moved
+ sleep 1
+ zpool import -d /dev/zvol/dsk/rpool/subvol1 pool
+ zpool status pool
  pool: pool
 state: ONLINE
 scrub: none requested
config:

NAME  STATE READ WRITE CKSUM
pool  ONLINE   0 0 0
  mirror-0ONLINE   0 0 0
/dev/zvol/dsk/rpool/subvol1/vol1  ONLINE   0 0 0
/dev/zvol/dsk/rpool/vol2  ONLINE   0 0 0

errors: No known data errors
### Note that vol2 has it old path shown!



### Interestingly, if you then
+ zpool export pool
+ zpool import -d /dev/zvol/dsk/rpool/subvol2 pool
### vol2's path gets updated too:
+ zpool status pool
  pool: pool
 state: ONLINE
 scrub: none requested
config:

NAME  STATE READ WRITE CKSUM
pool  ONLINE   0 0 0
  mirror-0ONLINE   0 0 0
/dev/zvol/dsk/rpool/subvol1/vol1  ONLINE   0 0 0
/dev/zvol/dsk/rpool/subvol2/vol2  ONLINE   0 0 0

errors: No known data errors



___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] How many TLDs should you have?

[Craig Cory]

Hi Chris,

It sounds like there is some confusion with the recommendation about raidz?
vdevs. It is recommended that each raidz? TLD be a "single-digit" number of
disks - so up to 9. The total number of these single digit TLDs is not
practically limited.

Craig



Christopher White wrote:
> Cindy --
>
> Thanks for the link!
>
> I see in one of the examples that there are 14 TLDs (all mirrored).  Does
> that mean there are no performance issues with having more than 9 TLDs?  In
> the Sun class I attended, the instructor said to not use more than 9 TLDs,
> which seems like it could be very limiting, especially in a SAN setting.
> Like I said, our storage group presents 15G LUNs to use -- so it'd be
> difficult to keep the TLDs under 9 and have a very large filesystem.
>
> Let me know what you think.  Thanks!
>
> Chris
>
>
>
> On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen
> wrote:
>
>> Hi Chris,
>>
>> If you have 40 or so disks then you would create 5-6 RAIDZ virtual
>> devices of 7-8 disks each, or possibly include two disks for the root
>> pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks
>> for a non-root pool.
>>
>> This configuration guide hasn't been updated for RAIDZ-3 yet, but you
>> will get some ideas about how to configure a redundant configuration
>> of many disks, here:
>>
>> http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide
>>
>> See ZFS Configuration Example (x4500 with raidz2)
>>
>> Cindy
>>
>>
>> On 12/01/09 09:20, Christopher White wrote:
>>
>>> All,
>>>
>>> We're going to start testing ZFS and I had a question about Top Level
>>> Devices (TLDs).  In Sun's class, they specifically said not to use more
>>> than
>>> 9 TLDs due to performance concerns.  Our storage admins make LUNs roughly
>>> 15G in size -- so how would we make a large pool (1TB) if we're limited to
>>> only 9 TLDs?
>>>
>>> The Best Practices guide (
>>> http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations)
>>> suggests not using 40+ disks in a RAIDZ TLD:
>>>
>>> "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration
>>> with one logical device of 40+ devices. See the sections below for examples
>>> of redundant configurations."
>>>
>>> Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each
>>> RAIDZ?
>>>
>>> Or is the 9 TLDs an old recommendation that has since been changed?
>>>
>>> Thanks!
>>>
>>> Chris
>>>
>>>
>>> 
>>>
>>> ___
>>> zfs-discuss mailing list
>>> zfs-discuss@opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>>
>>
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>


-- 
Craig Cory
 Senior Instructor :: ExitCertified
 : Sun Certified System Administrator
 : Sun Certified Network Administrator
 : Sun Certified Security Administrator
 : Veritas Certified Instructor

 8950 Cal Center Drive
 Bldg 1, Suite 110
 Sacramento, California  95826
 [e] craig.c...@exitcertified.com
 [p] 916.669.3970
 [f] 916.669.3977
 [w] WWW.EXITCERTIFIED.COM
+-+
   OTTAWA | SACRAMENTO | MONTREAL | LAS VEGAS | QUEBEC CITY | CALGARY
SAN FRANCISCO | VANCOUVER | REGINA | WINNIPEG | TORONTO

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Adding drives to system - disk labels not consistent

[Mark J. Musante]

This may be a dup of 6881631.



Regards,
markm

On 1 Dec 2009, at 15:14, Cindy Swearingen   
wrote:



I was able to reproduce this problem on the latest Nevada build:


# zpool create tank raidz c1t2d0 c1t3d0 c1t4d0
# zpool add -n tank raidz c1t5d0 c1t6d0 c1t7d0
would update 'tank' to the following configuration:
  tank
raidz1
  c1t2d0
  c1t3d0
  c1t4d0
raidz1
  c1t5
  c1t6
  c1t7

I will file a low-priority bug since it is a -n display problem
only.

Thanks for reporting it.

Cindy


On 11/30/09 18:15, Stuart Reid wrote:

Hi -
Using OpenSolaris 2008.11
Hope this is enough information.
Stuart
-Original Message-
From: cindy.swearin...@sun.com [mailto:cindy.swearin...@sun.com]
Sent: November 30, 2009 11:32 AM
To: Stuart Reid
Cc: zfs-discuss@opensolaris.org
Subject: Re: [zfs-discuss] Adding drives to system - disk labels not
consistent
Hi Stuart,
Which Solaris release are you seeing this behavior?
I would like reproduce it and file a bug, if necessary.
Thanks,
Cindy
On 11/29/09 13:06, Stuart Reid wrote:

Answered by own question...

When using the -n switch the output is truncated i.e. the d0 is not
printed. When actually adding the drives, the operation is as  
expected -

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss