Because this system was in production I had to fairly quickly recover, so I was unable to play much more with it we had to destroy it and recreate new pool and then recover data from tapes.
Its a mistery as to why in the middle of a night it rebooted, we could not figure this out and why pool had this problem... so unfortunatelly I will not be able to follow what you Victor and Jeff were suggesting. before we destroyed that pool I did get output of fmdump on that system to see what failed etc. As you can see it happend at around 3:54 am on Sunday morning there was no one on the system from admin perspective to break anything, only thing that I might think of would be the backups running which could generate more traffic, but then I had that system for over a year setup this way, and no changes were made to it from storage perspective. yes I did see this URL: http://www.opensolaris.org/jive/thread.jspa?messageID=220125 but unfortunately I was unable to apply it in my situation as I had no idea what values to apply... :( anyway here is fmdump bash-3.00# fmdump -eV TIME CLASS Apr 27 2008 03:54:05.605369200 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x18594234ea00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xc40696f31f78fd48 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xc40696f31f78fd48 vdev_type = disk vdev_path = /dev/dsk/emcpower0a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x4814311d 0x24153370 Apr 27 2008 03:54:05.605369725 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x18594234ea00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xd56fa2d7686dae8c (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xd56fa2d7686dae8c vdev_type = disk vdev_path = /dev/dsk/emcpower2a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x4814311d 0x2415357d Apr 27 2008 03:54:05.605369225 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x18594234ea00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 __ttl = 0x1 __tod = 0x4814311d 0x24153389 Apr 27 2008 03:56:28.180698100 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x398b69181e00401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xc40696f31f78fd48 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xc40696f31f78fd48 vdev_type = disk vdev_path = /dev/dsk/emcpower0a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x481431ac 0xac53bf4 Apr 27 2008 03:56:28.180698375 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x398b69181e00401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xd56fa2d7686dae8c (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xd56fa2d7686dae8c vdev_type = disk vdev_path = /dev/dsk/emcpower2a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x481431ac 0xac53d07 Apr 27 2008 03:56:28.180698500 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x398b69181e00401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 __ttl = 0x1 __tod = 0x481431ac 0xac53d84 Apr 28 2008 09:40:10.917082725 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x1fa8d23d6e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xc40696f31f78fd48 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xc40696f31f78fd48 vdev_type = disk vdev_path = /dev/dsk/emcpower0a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x4815d3ba 0x36a99265 Apr 28 2008 09:40:10.917081800 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x1fa8d23d6e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xd56fa2d7686dae8c (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xd56fa2d7686dae8c vdev_type = disk vdev_path = /dev/dsk/emcpower2a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x4815d3ba 0x36a98ec8 Apr 28 2008 09:40:10.917081825 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x1fa8d23d6e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 __ttl = 0x1 __tod = 0x4815d3ba 0x36a98ee1 Apr 28 2008 10:05:27.315099900 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x180ba4f67c600001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x4815d9a7 0x12c80afc Apr 28 2008 10:06:26.582074525 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x18e86f20a1a00401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x4815d9e2 0x22b1c09d Apr 28 2008 10:38:24.337459525 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0xc04d9834fc00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x4815e160 0x141d3945 Apr 28 2008 13:51:24.239005950 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0xb487e94711e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x48160e9c 0xe3ef0fe Apr 28 2008 14:00:20.762159250 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0xbc56a37469c00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x481610b4 0x2d6da092 Apr 28 2008 15:10:46.512345000 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0xf9d4f37becc00401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 2 __ttl = 0x1 __tod = 0x48162136 0x1e89c3a8 Apr 29 2008 11:10:49.126505525 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x2232b54d9e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xc40696f31f78fd48 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xc40696f31f78fd48 vdev_type = disk vdev_path = /dev/dsk/emcpower0a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x48173a79 0x78a5235 Apr 29 2008 11:10:49.126505300 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x2232b54d9e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0xd56fa2d7686dae8c (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0xd56fa2d7686dae8c vdev_type = disk vdev_path = /dev/dsk/emcpower2a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x48173a79 0x78a5154 Apr 29 2008 11:10:49.126505475 ereport.fs.zfs.vdev.open_failed nvlist version: 0 class = ereport.fs.zfs.vdev.open_failed ena = 0x2232b54d9e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 vdev = 0x867a58f8e0043ecd (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 vdev_guid = 0x867a58f8e0043ecd vdev_type = disk vdev_path = /dev/dsk/emcpower3a parent_guid = 0x39918ce32491d000 parent_type = root prev_state = 0x1 __ttl = 0x1 __tod = 0x48173a79 0x78a5203 Apr 29 2008 11:10:49.126504575 ereport.fs.zfs.zpool nvlist version: 0 class = ereport.fs.zfs.zpool ena = 0x2232b54d9e00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x39918ce32491d000 (end detector) pool = mypool pool_guid = 0x39918ce32491d000 pool_context = 1 __ttl = 0x1 __tod = 0x48173a79 0x78a4e7f bash-3.00# On Sun, 4 May 2008, Victor Latushkin wrote: > Jeff Bonwick wrote: >>> Looking at the txg numbers, it's clear that labels on to devices that >>> are unavailable now may be stale: >> >> Actually, they look OK. The txg values in the label indicate the >> last txg in which the pool configuration changed for devices in that >> top-level vdev (e.g. mirror or raid-z group), not the last txg synced. > > Agree, I've jumped to conclusions here. > > But still it is a difference between two labels presented. Since this was > running for a while I suppose there have been no admin-initiated > configuration changes, so config change may be due to allocation of DTL > object, correct? > > Still it would be interesting to know txg of the selected uberblock to see > how long ago that change happened. > > Also it would be interesting to know why did server reboot? > > Victor > > > !DSPAM:122,481d7f2e45852151120594! > _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
