Hello, Has anyone ever seen vdev's getting removed and added back to the pool very quickly ? That seems to be what's happening here.
This has started to happen on dozens of machines at different locations since a few days ago. They are running OpenSolaris b111 and a few b126. Could this be bit rot and/or silent corruption getting detected and fixed ? Jan 27 01:18:01 hostname fmd: [ID 441519 daemon.notice] SUNW-MSG-ID: FMD-8000-4M, TYPE: Repair, VER: 1, SEVERITY: Minor Jan 27 01:18:01 hostname EVENT-TIME: Thu Dec 24 08:50:34 BRST 2009 Jan 27 01:18:01 hostname PLATFORM: X7DB8, CSN: 0123456789, HOSTNAME: hostname Jan 27 01:18:01 hostname SOURCE: fmd, REV: 1.2 Jan 27 01:18:01 hostname EVENT-ID: 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd Jan 27 01:18:01 hostname DESC: All faults associated with an event id have been addressed. Jan 27 01:18:01 hostname Refer to http://sun.com/msg/FMD-8000-4M for more information. Jan 27 01:18:01 hostname AUTO-RESPONSE: Some system components offlined because of the original fault may have been brought back online. Jan 27 01:18:01 hostname IMPACT: Performance degradation of the system due to the original fault may have been recovered. Jan 27 01:18:01 hostname REC-ACTION: Use fmdump -v -u <EVENT-ID> to identify the repaired components. Jan 27 01:18:01 hostname fmd: [ID 441519 daemon.notice] SUNW-MSG-ID: FMD-8000-6U, TYPE: Resolved, VER: 1, SEVERITY: Minor Jan 27 01:18:01 hostname EVENT-TIME: Thu Dec 24 08:50:34 BRST 2009 Jan 27 01:18:01 hostname PLATFORM: X7DB8, CSN: 0123456789, HOSTNAME: hostname Jan 27 01:18:01 hostname SOURCE: fmd, REV: 1.2 Jan 27 01:18:01 hostname EVENT-ID: 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd Jan 27 01:18:01 hostname DESC: All faults associated with an event id have been addressed. Jan 27 01:18:01 hostname Refer to http://sun.com/msg/FMD-8000-6U for more information. Jan 27 01:18:01 hostname AUTO-RESPONSE: All system components offlined because of the original fault have been brought back online. Jan 27 01:18:01 hostname IMPACT: Performance degradation of the system due to the original fault has been recovered. Jan 27 01:18:01 hostname REC-ACTION: Use fmdump -v -u <EVENT-ID> to identify the repaired components. # fmdump -e -t 23Jan2010 TIME CLASS # # fmdump TIME UUID SUNW-MSG-ID Jan 27 01:18:01.2372 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd FMD-8000-4M Repaired Jan 27 01:18:01.2391 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd FMD-8000-6U Resolved # fmdump -V TIME UUID SUNW-MSG-ID Jan 27 01:18:01.2372 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd FMD-8000-4M Repaired TIME CLASS ENA Dec 24 08:50:34.4470 ereport.fs.zfs.vdev.corrupt_data 0x533bf0e964a01801 Dec 23 16:08:42.0738 ereport.fs.zfs.probe_failure 0xe87b448c8ba00c01 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b446b04f00001 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b44664b300401 Dec 23 16:08:42.0738 ereport.fs.zfs.io 0xe87b445710a01001 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b4461a4d00c01 nvlist version: 0 version = 0x0 class = list.repaired uuid = 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd code = FMD-8000-4M diag-time = 1261651834 766268 de = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = fmd authority = (embedded nvlist) nvlist version: 0 version = 0x0 product-id = X7DB8 chassis-id = 0123456789 server-id = hostname (end authority) mod-name = fmd mod-version = 1.2 (end de) fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = fault.fs.zfs.device certainty = 0x64 asru = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x9f4842f183c4c7cc vdev = 0xd207014426714df9 (end asru) resource = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x9f4842f183c4c7cc vdev = 0xd207014426714df9 (end resource) (end fault-list[0]) fault-status = 0x6 __ttl = 0x1 __tod = 0x4b5fb069 0xe23eb38 TIME UUID SUNW-MSG-ID Jan 27 01:18:01.2391 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd FMD-8000-6U Resolved TIME CLASS ENA Dec 24 08:50:34.4470 ereport.fs.zfs.vdev.corrupt_data 0x533bf0e964a01801 Dec 23 16:08:42.0738 ereport.fs.zfs.probe_failure 0xe87b448c8ba00c01 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b446b04f00001 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b44664b300401 Dec 23 16:08:42.0738 ereport.fs.zfs.io 0xe87b445710a01001 Dec 23 16:08:42.0739 ereport.fs.zfs.io 0xe87b4461a4d00c01 nvlist version: 0 version = 0x0 class = list.resolved uuid = 0cb73c5a-d444-ede6-e49f-fce4aad8a1cd code = FMD-8000-6U diag-time = 1261651834 766268 de = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = fmd authority = (embedded nvlist) nvlist version: 0 version = 0x0 product-id = X7DB8 chassis-id = 0123456789 server-id = hostname (end authority) mod-name = fmd mod-version = 1.2 (end de) fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = fault.fs.zfs.device certainty = 0x64 asru = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x9f4842f183c4c7cc vdev = 0xd207014426714df9 (end asru) resource = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0x9f4842f183c4c7cc vdev = 0xd207014426714df9 (end resource) (end fault-list[0]) fault-status = 0x6 __ttl = 0x1 __tod = 0x4b5fb069 0xe411fc8 Thanks, -- Giovanni _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
