I'm cross-posting to zfs-discuss, as this is now more of a ZFS
query than a dtrace query at this point, and I'm not sure if all the ZFS
experts are listening on dtrace-discuss (although they probably
are... :^).
The only thing that jumps out at me is the ARC size - 53.4GB, or
most of your 64GB of RAM. This in-and-of-itself is not necessarily
a bad thing - if there are no other memory consumers, let ZFS cache
data in the ARC. But if something is coming along to flush dirty
ARC pages periodically....
I found what looked to be an applicable bug;
CR 6699438 zfs induces crosscall storm under heavy mapped sequential
read workload
but the stack signature for the above bug is different than yours, and
it doesn't sound like your workload is doing mmap'd sequential reads.
That said, I would be curious to know if your workload used mmap(),
versus read/write?
For the ZFS folks just seeing this, here's the stack frame;
unix`xc_do_call+0x8f
unix`xc_wait_sync+0x36
unix`x86pte_invalidate_pfn+0x135
unix`hat_pte_unmap+0xa9
unix`hat_unload_callback+0x109
unix`hat_unload+0x2a
unix`segkmem_free_vn+0x82
unix`segkmem_zio_free+0x10
genunix`vmem_xfree+0xee
genunix`vmem_free+0x28
genunix`kmem_slab_destroy+0x80
genunix`kmem_slab_free+0x1be
genunix`kmem_magazine_destroy+0x54
genunix`kmem_depot_ws_reap+0x4d
genunix`taskq_thread+0xbc
unix`thread_start+0x8
Let's see what the fsstat and zpool iostat data looks like when this
starts happening..
Thanks,
/jim
Jim Leonard wrote:
It would also be interesting to see some snapshots
of the ZFS arc kstats
kstat -n arcstats
Here you go, although I didn't see anything jump out (massive amounts of cache
misses or something). Any immediate trouble spot?
# kstat -n arcstats
module: zfs instance: 0
name: arcstats class: misc
c 53490612870
c_max 67636535296
c_min 8454566912
crtime 212.955493179
deleted 7267003
demand_data_hits 179708165
demand_data_misses 189797
demand_metadata_hits 9959277
demand_metadata_misses 194228
evict_skip 1709
hash_chain_max 9
hash_chains 205513
hash_collisions 9372169
hash_elements 851634
hash_elements_max 886509
hdr_size 143082240
hits 198822346
l2_abort_lowmem 0
l2_cksum_bad 0
l2_evict_lock_retry 0
l2_evict_reading 0
l2_feeds 0
l2_free_on_write 0
l2_hdr_size 0
l2_hits 0
l2_io_error 0
l2_misses 0
l2_rw_clash 0
l2_size 0
l2_writes_done 0
l2_writes_error 0
l2_writes_hdr_miss 0
l2_writes_sent 0
memory_throttle_count 0
mfu_ghost_hits 236508
mfu_hits 165895558
misses 388618
mru_ghost_hits 70149
mru_hits 24777390
mutex_miss 6094
p 49175731760
prefetch_data_hits 7993639
prefetch_data_misses 370
prefetch_metadata_hits 1161265
prefetch_metadata_misses 4223
recycle_miss 7149
size 53490565328
snaptime 5759009.53378144
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
