These stack traces look like 6569719 (fixed in s10u5).
For update 5, you could start with the kernel stack of the hung commands.
(use ::pgrep and ::findstack) We might also need the sync thread's stack
(something like ::walk spa | ::print spa_t
spa_dsl_pool->dp_txg.tx_sync_thread | ::findstack)
--matt
Andreas Koppenhoefer wrote:
>> Could you provide the panic message and stack trace,
>> plus the stack traces of when it's hung?
>>
>> --matt
>
> Hello matt,
>
> here is info and stack trace of a server running Update 3:
>
> $ uname -a
> SunOS qacpp03 5.10 Generic_127111-05 sun4us sparc FJSV,GPUSC-M
> $ head -1 /etc/release
> Solaris 10 11/06 s10s_u3wos_10 SPARC
> $ grep panic: /var/adm/messages.0
> Nov 4 16:04:42 qacpp03 savecore: [ID 570001 auth.error] reboot after panic:
> dangling dbufs (dn=600ac879410, dbuf=600475682c0)
> $ mdb -k unix.4 vmcore.4
> Loading modules: [ unix krtld genunix specfs dtrace ufs sd pcipsy ssd fcp
> fctl md emlxs ip hook neti sctp arp usba nca random zfs nfs crypto sppp ipc
> logindmux ptm cpc fcip lofs ]
>> $C
> 000002a103108811 vpanic(7b2e1df0, 600ac879410, 600475682c0, 600ac879638,
> 60047568350, 90)
> 000002a1031088c1 dnode_evict_dbufs+0x19c(600ac879410, 1, 7b2e1c00,
> 6000b0be500, 1, 2a103109170)
> 000002a103108a31 dmu_objset_evict_dbufs+0xd8(17, 0, 0, 7b2db400, 6000b0be380,
> 6000b0be380)
> 000002a103108ae1 dmu_objset_evict+0xb4(600054f9480, 6000b0be380, 7b2e1800,
> 7095b4f0, 7b2e1800, 7095b400)
> 000002a103108ba1 dsl_dataset_evict+0x34(600054f9480, 600054f9480,
> deadbeef00000000, 6000b0be380, 60076b3a700, 6009b489200)
> 000002a103108c51 dbuf_evict_user+0x48(7b2a3f00, 6009b489200, 600054f9488, 0,
> 0, 6009a99b5c0)
> 000002a103108d01 dbuf_rele+0x8c(6009a99b618, 6009a99b5c0, 70954400, 3, 3, 3)
> 000002a103108db1 dmu_recvbackup+0x94c(60046a32a80, 60046a33400, 60046a332a8,
> 60046a32a96, 16, 1)
> 000002a103108f81 zfs_ioc_recvbackup+0x74(60046a32680, 0, 6003b454130, 0, 0,
> 60046a32a80)
> 000002a103109041 zfsdev_ioctl+0x15c(70955400, 57, ffbfee28, 1d, 74, ef0)
> 000002a1031090f1 fop_ioctl+0x20(6000a020b80, 5a1d, ffbfee28, 100003,
> 6003b28c020, 121e728)
> 000002a1031091a1 ioctl+0x184(3, 6003b159680, ffbfee28, 0, 1ec08, 5a1d)
> 000002a1031092f1 syscall_trap32+0xcc(3, 5a1d, ffbfee28, 0, 1ec08, ff347734)
> ----------------------------------------------
>
> Panic of server running Update 4:
>
> $ uname -a
> SunOS vs104is1 5.10 Generic_127112-02 i86pc i386 i86pc
> $ head -1 /etc/release
> Solaris 10 8/07 s10x_u4wos_12b X86
> $ grep panic: /var/adm/messages.0
> Nov 4 14:37:15 vs104is1 savecore: [ID 570001 auth.error] reboot after panic:
> dangling dbufs (dn=d5662518, dbuf=ded05420)
> $ mdb -k unix.0 vmcore.0
> Loading modules: [ unix krtld genunix specfs dtrace uppc pcplusmp ufs ip hook
> neti sctp arp usba fctl nca lofs zfs random nfs sppp ptm ]
>> $C
> e5f73b68 vpanic(f9cdf0e4, d5662518, ded05420)
> e5f73c08 dnode_evict_dbufs+0x143(d5662518, 0)
> e5f73c28 dmu_objset_evict_dbufs+0xe3(e5f73c40, 0)
> e5f73c4c dmu_objset_evict+0x72(eab53e40, d56f8940)
> e5f73c68 dsl_dataset_evict+0x2f(f16fc240, eab53e40)
> e5f73c7c dbuf_evict_user+0x29(f16fc240)
> e5f73c9c dbuf_rele+0x7a(f16fc240, f9cd9519)
> e5f73cbc dsl_dataset_close+0x57(eab53e40, 3, f9cd9519)
> e5f73e0c dmu_recvbackup+0x88f(dec54400, dec54d80, dec54c28, 0, d6d48180, 186)
> e5f73e58 zfs_ioc_recvbackup+0x85(dec54000)
> e5f73e78 zfsdev_ioctl+0xf8()
> e5f73e98 cdev_ioctl+0x2b(2d40000, 5a1d, 804680c, 100003, ee43fb60, e5f73f78)
> e5f73ebc spec_ioctl+0x62(d689d900, 5a1d, 804680c, 100003, ee43fb60, e5f73f78)
> e5f73eec fop_ioctl+0x24(d689d900, 5a1d, 804680c, 100003, ee43fb60, e5f73f78)
> e5f73f84 ioctl+0x199()
> e5f73fac sys_sysenter+0x101()
> -------------------------------
> Hope this helps.
>
> Unfortunately I have no idea which kind of stack trace you need of the server
> running Update 5!?
> A stack trace of the running kernel is useless, since it will no show any
> usefull informationen. It's the zfs command ("zfs receive" and "zfs list") in
> user space which are hanging.
> What kind of inforation do you need for analysis and how can I provide them?
> Or did I miss something?
>
> - Andreas
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
