Hello,
I'd like a sanity check from people more knowledgeable than myself.
I'm managing backups on a production system. Previously I was using
another volume manager and filesystem on Solaris, and I've just switched
to using ZFS.
My model is -
Production Server A
Test Server B
Mirrored storage arrays (HDS TruCopy if it matters)
Backup software (TSM)
Production server A sees the live volumes.
Test Server B sees the TruCopy mirrors of the live volumes. (it sees
the second storage array, the production server sees the primary array)
Production server A shuts down zone C, and exports the zpools for zone C.
Production server A splits the mirror to secondary storage array,
leaving the mirror writable.
Production server A re-imports the pools for zone C, and boots zone C.
Test Server B imports the ZFS pool using -R /backup.
Backup software backs up the mounted mirror volumes on Test Server B.
Later in the day after the backups finish, a script exports the ZFS
pools on test server B, and re-establishes the TruCopy mirror between
the storage arrays.
So.. I had this working fine with one zone on server A for a couple of
months. This week I've added 6 more zones, each with two ZFS pools.
The first night went okay. Last night, the test server B kernel panic'd
well after the mirrored volumes zpools were imported, just after the TSM
backup started reading all the ZFS pools to push it all to the
enterprise backup environment.
Here's the kernel panic message -
Jul 6 03:04:55 riggs ^Mpanic[cpu22]/thread=2a10e81bca0:
Jul 6 03:04:55 riggs unix: [ID 403854 kern.notice] assertion failed: 0
== dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs,
&dbp), file: ../../common/fs/zfs/dmu.c, line: 759
Jul 6 03:04:55 riggs unix: [ID 100000 kern.notice]
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b4f0
genunix:assfail+74 (7af0f8c0, 7af0f910, 2f7, 190d000, 12a1800, 0)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 0000000000000001 0000000000000001 00000300f20fdf81
Jul 6 03:04:55 riggs %l4-7: 00000000012a1800 0000000000000000
0000000001959400 0000000000000000
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b5a0
zfs:dmu_write+54 (300cbfd5c40, ad, a70, 20, 300b8c02800, 300f83414d0)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000038 0000000000000007 000000000194bd40 000000000194bc00
Jul 6 03:04:55 riggs %l4-7: 0000000000000001 0000030071bcb701
0000000000003006 0000000000003000
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b670
zfs:space_map_sync+278 (3009babd130, b, 3009babcfe0, 20, 4, 58)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000020 00000300b8c02800 00000300b8c02820 00000300b8c02858
Jul 6 03:04:55 riggs %l4-7: 00007fffffffffff 0000000000007fff
00000000000022d9 0000000000000020
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b760
zfs:metaslab_sync+2b0 (3009babcfc0, 1db7, 300f83414d0, 3009babd408,
300c9724000, 6003e24acc0)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
00000300cbfd5c40 000003009babcff8 000003009babd130 000003009babd2d0
Jul 6 03:04:55 riggs %l4-7: 000003009babcfe0 0000000000000000
000003009babd268 000000000000001a
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b820
zfs:vdev_sync+b8 (6003e24acc0, 1db7, 1db6, 3009babcfc0, 6003e24b000, 17)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000090 0000000000000012 000006003e24acc0 00000300c9724000
Jul 6 03:04:55 riggs %l4-7: 0000000000000000 0000000000000000
0000000000000000 00000009041ea000
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b8d0
zfs:spa_sync+484 (300c9724000, 1db7, 3005fec09a8, 300c9724428, 1,
300cbfd5c40)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 00000300c9724280 0000030087c3e940 00000300c6aae700
Jul 6 03:04:55 riggs %l4-7: 0000030080073520 00000300c9724378
00000300c9724300 00000300c9724330
Jul 6 03:04:55 riggs genunix: [ID 723222 kern.notice] 000002a10e81b9a0
zfs:txg_sync_thread+1b8 (30087c3e940, 183f9f0, 707a3130, 0, 2a10e81ba70, 0)
Jul 6 03:04:55 riggs genunix: [ID 179002 kern.notice] %l0-3:
0000000000000000 0000030087c3eb0e 0000030087c3eb08 0000030087c3eb0c
Jul 6 03:04:55 riggs %l4-7: 000000001230fa07 0000030087c3eac8
0000030087c3ead0 0000000000001db7
Jul 6 03:04:55 riggs unix: [ID 100000 kern.notice]
So, I guess my question is - is what I'm doing sane? Or is there
something inherint with ZFS that I'm missing that's going to cause this
kernel panic to repeat? Best I can guess, it got upset when the pools
were being read. I'm wondering of exporting the pools later in the day
before re-syncing the SAN volumes to mirrors is causing weirdness.
(because makes the mirrored volumes visible on Test Server B read-only
until the split). I wouldn't think so, because they're exported before
the luns go read-only, but I could be wrong.
Anyway, am I off my rocker? This should work with ZFS, right?
Thanks!
Brian
--
-----------------------------------------------------------------------------------
Brian Wilson, Solaris SE, UW-Madison DoIT
Room 3114 CS&S 608-263-8047
brian.wilson(a)doit.wisc.edu
'I try to save a life a day. Usually it's my own.' - John Crichton
-----------------------------------------------------------------------------------
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
