Ok, here is the scoop on the dire Solaris 10 (Generic_141415-03)
performance bug on my Sun Ultra 40-M2 attached to a StorageTek 2540
with latest firmware. I rebooted the system used cpio to send the
input files to /dev/null, and then immediately used cpio a second time
to send the input files to /dev/null. Note that the amount of file
data (243 GB) is plenty sufficient to purge any file data from the ARC
(which has a cap of 10 GB).
% time cat dpx-files.txt | cpio -o > /dev/null
495713288 blocks
cat dpx-files.txt 0.00s user 0.00s system 0% cpu 1.573 total
cpio -o > /dev/null 78.92s user 360.55s system 43% cpu 16:59.48 total
% time cat dpx-files.txt | cpio -o > /dev/null
495713288 blocks
cat dpx-files.txt 0.00s user 0.00s system 0% cpu 0.198 total
cpio -o > /dev/null 79.92s user 358.75s system 11% cpu 1:01:05.88 total
zpool iostat averaged over 60 seconds reported that the first run
through the files read the data at 251 MB/s and the second run only
achieved 68 MB/s. It seems clear that there is something really bad
about Solaris 10 zfs's file caching code which is causing it to go
into the weeds.
I don't think that the results mean much, but I have attached output
from 'hotkernel' while a subsequent cpio copy is taking place. It
shows that the kernel is mostly sleeping.
This is not a new problem. It seems that I have been banging my head
against this from the time I started using zfs.
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Sampling... Hit Ctrl-C to end.
FUNCTION COUNT PCNT
unix`SHA1Update 1 0.0%
unix`page_unlock 1 0.0%
unix`lwp_segregs_save 1 0.0%
rootnex`rootnex_dma_allochdl 1 0.0%
unix`mutex_delay_default 1 0.0%
emlxs`emlxs_initialize_pkt 1 0.0%
genunix`pid_lookup 1 0.0%
TS`ts_setrun 1 0.0%
fcp`ssfcp_adjust_cmd 1 0.0%
genunix`strrput 1 0.0%
genunix`cyclic_softint 1 0.0%
genunix`fop_poll 1 0.0%
sd`sd_xbuf_strategy 1 0.0%
ohci`ohci_state_is_operational 1 0.0%
zfs`SHA256Transform 1 0.0%
unix`cpu_resched 1 0.0%
nvidia`_nv006110rm 1 0.0%
genunix`lwp_timer_timeout 1 0.0%
genunix`realtime_timeout 1 0.0%
fcp`ssfcp_scsi_destroy_pkt 1 0.0%
nvidia`nvidia_pci_check_config_space 1 0.0%
genunix`closef 1 0.0%
sd`sd_setup_rw_pkt 1 0.0%
unix`vsnprintf 1 0.0%
zfs`vdev_dtl_contains 1 0.0%
genunix`siginfo_kto32 1 0.0%
iommulib`iommulib_nex_open 1 0.0%
genunix`vn_has_cached_data 1 0.0%
ohci`ohci_sendup_td_message 1 0.0%
scsi_vhci`vhci_scsi_destroy_pkt 1 0.0%
genunix`avl_add 1 0.0%
unix`page_create_va 1 0.0%
genunix`savectx 1 0.0%
ohci`ohci_root_hub_allocate_intr_pipe_resource 1 0.0%
unix`page_add 1 0.0%
zfs`zfs_unix_to_v4 1 0.0%
genunix`set_qend 1 0.0%
zfs`vdev_queue_io_done 1 0.0%
unix`set_idle_cpu 1 0.0%
zfs`vdev_cache_read 1 0.0%
nvidia`_nv002998rm 1 0.0%
ohci`ohci_do_intrs_stats 1 0.0%
genunix`putq 1 0.0%
genunix`strput 1 0.0%
zfs`zio_buf_alloc 1 0.0%
sockfs`socktpi_poll 1 0.0%
sockfs`so_update_attrs 1 0.0%
sockfs`so_unlock_read 1 0.0%
zfs`zfs_zaccess 1 0.0%
genunix`schedctl_save 1 0.0%
nvidia`_nv004051rm 1 0.0%
zfs`dbuf_destroy 1 0.0%
nvidia`_nv001416rm 1 0.0%
genunix`ddi_dma_buf_bind_handle 1 0.0%
zfs`zio_decompress_data 1 0.0%
genunix`bdev_strategy 1 0.0%
kcf`hmac_encr 1 0.0%
unix`page_trylock 1 0.0%
unix`hat_page_getattr 1 0.0%
genunix`fop_open 1 0.0%
zfs`zfs_zaccess_common 1 0.0%
emlxs`emlxs_sli2_bde_setup 1 0.0%
genunix`sprintf 1 0.0%
unix`ddi_get8 1 0.0%
zfs`zfs_dirlook 1 0.0%
zfs`zio_clear_transform_stack 1 0.0%
genunix`untimeout 1 0.0%
genunix`allocb_cred 1 0.0%
unix`lock_clear 1 0.0%
iommulib`lookup_cache 1 0.0%
zfs`dbuf_hash_insert 1 0.0%
genunix`syscall_exit 1 0.0%
unix`strcmp 1 0.0%
unix`hr_clock_lock 1 0.0%
genunix`delfpollinfo 1 0.0%
unix`i_ddi_caut_get16 1 0.0%
genunix`fop_access 1 0.0%
unix`getflags 1 0.0%
zfs`arc_cksum_verify 1 0.0%
zfs`vdev_disk_io_done 1 0.0%
zfs`spa_config_enter 1 0.0%
emlxs`emlxs_thread_trigger2 1 0.0%
TS`ts_preempt 1 0.0%
ldterm`ldtermrput 1 0.0%
usba`usba_vlog 1 0.0%
genunix`crgetuid 1 0.0%
unix`idle_enter 1 0.0%
genunix`sleepq_unsleep 1 0.0%
unix`mutex_vector_enter 1 0.0%
unix`lgrp_loadavg 1 0.0%
kcf`kcf_rnd_get_pseudo_bytes 1 0.0%
sd`sd_fill_scsi1_lun 1 0.0%
nge`nge_hot_rxd_fill 1 0.0%
genunix`vmem_xalloc 1 0.0%
nvidia`_nv006118rm 1 0.0%
TS`ts_update_list 1 0.0%
nvidia`_nv002896rm 1 0.0%
genunix`pn_setlast 1 0.0%
genunix`pcache_poll 1 0.0%
iommulib`hashfn 1 0.0%
ldterm`newmsg 1 0.0%
genunix`signal_is_blocked 1 0.0%
genunix`kmem_slab_alloc 1 0.0%
unix`i_ddi_caut_get32 1 0.0%
unix`xcopyout 1 0.0%
zfs`dmu_object_size_from_db 1 0.0%
tl`tl_wput_data_ser 1 0.0%
genunix`disp_lock_enter 1 0.0%
genunix`bioerror 1 0.0%
dtrace`dtrace_ioctl 1 0.0%
nvidia`_nv008153rm 1 0.0%
genunix`avl_first 1 0.0%
zfs`zio_read_decompress 1 0.0%
unix`pci_peekpoke_check 1 0.0%
emlxs`emlxs_proc_ring_event 1 0.0%
genunix`list_insert_head 1 0.0%
genunix`tsd_get 1 0.0%
nvidia`_nv006127rm 1 0.0%
zfs`zfs_ace_v0_get_mask 1 0.0%
genunix`pcache_insert 1 0.0%
genunix`crhold 1 0.0%
zfs`zio_should_retry 1 0.0%
zfs`arc_adapt 1 0.0%
specfs`spec_ioctl 1 0.0%
ohci`ohci_deallocate_tw_resources 1 0.0%
genunix`restorecontext 1 0.0%
genunix`freeb 1 0.0%
genunix`timespecfix 1 0.0%
genunix`ddi_dma_unbind_handle 1 0.0%
genunix`vmem_size 1 0.0%
genunix`schedctl_sigblock 1 0.0%
dtrace`dtrace_speculation_clean 1 0.0%
scsi`scsi_destroy_pkt 1 0.0%
genunix`new_mstate 1 0.0%
nvidia`_nv006129rm 1 0.0%
zfs`dbuf_set_data 1 0.0%
unix`i_ddi_caut_getput_ctlops 1 0.0%
fctl`fc_ulp_init_packet 1 0.0%
genunix`ddi_dma_freehdl 1 0.0%
nvidia`_nv012491rm 1 0.0%
unix`pc_tod_get 1 0.0%
tl`tl_wput 1 0.0%
zfs`arc_buf_destroy 1 0.0%
ehci`ehci_get_root_hub_port_status 1 0.0%
nvidia`_nv012492rm 1 0.0%
nvidia`_nv004854rm 1 0.0%
genunix`copy_pattern 1 0.0%
unix`av_check_softint_pending 1 0.0%
zfs`rrw_exit 1 0.0%
rootnex`rootnex_clean_dmahdl 1 0.0%
genunix`mod_major_to_name 1 0.0%
genunix`ddi_dma_allochdl 1 0.0%
genunix`strdoioctl 1 0.0%
nvidia`_nv004086rm 1 0.0%
genunix`cv_signal 1 0.0%
genunix`ddi_ctlops 1 0.0%
genunix`ldi_strategy 1 0.0%
nvidia`_nv002887rm 1 0.0%
zfs`zfs_fuid_map_ids 1 0.0%
genunix`schedctl_restore 1 0.0%
genunix`mp_cont_len 1 0.0%
zfs`arc_reclaim_needed 1 0.0%
ip`tcp_send_find_ire 1 0.0%
genunix`setrun 1 0.0%
genunix`kmem_cpu_reload 1 0.0%
genunix`strget 1 0.0%
nvidia`_nv006152rm 1 0.0%
unix`hat_kpm_pfn2va 1 0.0%
genunix`seg_ppurge_all 1 0.0%
genunix`timespectohz 1 0.0%
nvidia`_nv008179rm 1 0.0%
genunix`disp_lock_enter_high 1 0.0%
genunix`clear_stale_fd 1 0.0%
zfs`zio_nowait 1 0.0%
genunix`setf 1 0.0%
genunix`pcacheset_resolve 2 0.0%
unix`set_freemem 2 0.0%
unix`hr_clock_unlock 2 0.0%
fctl`fc_ulp_uninit_packet 2 0.0%
emlxs`emlxs_tx_get 2 0.0%
nvidia`_nv004781rm 2 0.0%
unix`kstat_runq_exit 2 0.0%
zfs`dbuf_findbp 2 0.0%
nvidia`_nv003170rm 2 0.0%
unix`prefetch_page_r 2 0.0%
sd`sd_buf_iodone 2 0.0%
genunix`clock 2 0.0%
unix`x86pte_release_pagetable 2 0.0%
rootnex`rootnex_dma_bindhdl 2 0.0%
nvidia`_nv008946rm 2 0.0%
scsi`scsi_transport 2 0.0%
zfs`vdev_cache_lastused_compare 2 0.0%
nvidia`rm_isr 2 0.0%
pcplusmp`apic_redistribute_compute 2 0.0%
zfs`spa_get_random 2 0.0%
emlxs`emlxs_mem_put 2 0.0%
usba`usba_rm_from_list 2 0.0%
unix`pg_cmt_load 2 0.0%
zfs`dbuf_hold 2 0.0%
zfs`vdev_queue_io 2 0.0%
zfs`zio_vdev_io_assess 2 0.0%
unix`atomic_inc_64 2 0.0%
zfs`zio_vdev_io_start 2 0.0%
zfs`lzjb_compress 2 0.0%
unix`mutex_vector_tryenter 2 0.0%
genunix`kmem_depot_free 2 0.0%
nvidia`_nv003008rm 2 0.0%
unix`lock_clear_splx 2 0.0%
genunix`ddi_dma_alloc_handle 2 0.0%
nvidia`_nv004840rm 2 0.0%
unix`lock_set_spl_spin 2 0.0%
genunix`canputnext 2 0.0%
zfs`arc_get_data_buf 2 0.0%
unix`tlb_service 2 0.0%
fcp`ssfcp_transport 2 0.0%
zfs`dnode_block_freed 2 0.0%
fcp`ssfcp_complete_pkt 2 0.0%
emlxs`emlxs_timer_check_pkts 2 0.0%
specfs`spec_poll 2 0.0%
zfs`dbuf_update_data 2 0.0%
genunix`ddi_driver_major 2 0.0%
genunix`thread_free_prevent 2 0.0%
genunix`strioctl 2 0.0%
genunix`allocb 2 0.0%
genunix`pcacheset_cmp 2 0.0%
genunix`biodone 2 0.0%
genunix`ddi_get_soft_state 2 0.0%
emlxs`emlxs_swap_fcp_pkt 2 0.0%
genunix`list_head 2 0.0%
zfs`arc_access 2 0.0%
zfs`zfs_dirent_lock 2 0.0%
unix`drv_usecwait 2 0.0%
genunix`restorectx 2 0.0%
genunix`getq_noenab 2 0.0%
unix`kstat_waitq_to_runq 2 0.0%
genunix`clock_tick 2 0.0%
genunix`mdi_pi_get_vhci_private 2 0.0%
scsi_vhci`vhci_scsi_start 2 0.0%
zfs`vdev_readable 2 0.0%
fcp`ssfcp_init_ventilators 2 0.0%
genunix`sleepq_dequeue 2 0.0%
genunix`timeout 2 0.0%
unix`mutex_adaptive_tryenter 2 0.0%
nvidia`_nv008401rm 2 0.0%
scsi`scsi_hba_pkt_alloc 2 0.0%
rootnex`rootnex_coredma_bindhdl 2 0.0%
emlxs`emlxs_msi_intr 2 0.0%
unix`clock_tick_schedule 2 0.0%
nvidia`_nv002782rm 3 0.0%
ohci`ohci_handle_root_hub_status_change 3 0.0%
zfs`arc_buf_freeze 3 0.0%
emlxs`emlxs_sli3_bde_setup 3 0.0%
fctl`fc_ulp_transport 3 0.0%
genunix`set_anoninfo 3 0.0%
nvidia`_nv005978rm 3 0.0%
zfs`add_reference 3 0.0%
zfs`dnode_hold 3 0.0%
zfs`zio_destroy 3 0.0%
zfs`zfs_range_unlock_reader 3 0.0%
genunix`new_cpu_mstate 3 0.0%
genunix`cv_timedwait_sig 3 0.0%
genunix`kmem_alloc 3 0.0%
unix`ddi_io_put32 3 0.0%
unix`disp_lowpri_cpu 3 0.0%
sockfs`socktpi_ioctl 3 0.0%
genunix`uiomove 3 0.0%
genunix`disp_lock_exit_high 3 0.0%
zfs`dbuf_do_evict 3 0.0%
zfs`buf_hash_insert 3 0.0%
nvidia`_nv001414rm 3 0.0%
unix`clock_tick_execute_common 3 0.0%
unix`mutex_destroy 3 0.0%
unix`clock_tick_process 3 0.0%
unix`fpxsave_begin 3 0.0%
zfs`vdev_lookup_top 3 0.0%
zfs`zio_wait_for_children_ready 3 0.0%
genunix`pollsys 3 0.0%
genunix`fop_ioctl 3 0.0%
genunix`avl_insert 3 0.0%
rootnex`rootnex_coredma_unbindhdl 3 0.0%
genunix`kmem_zalloc 3 0.0%
unix`lwp_segregs_restore32 3 0.0%
rootnex`rootnex_dma_unbindhdl 4 0.0%
zfs`vdev_stat_update 4 0.0%
genunix`ddi_get_instance 4 0.0%
unix`av_dispatch_softvect 4 0.0%
zfs`rrw_enter_read 4 0.0%
emlxs`emlxs_timer 4 0.0%
fcp`ssfcp_scsi_start 4 0.0%
zfs`remove_reference 4 0.0%
genunix`cv_waituntil_sig 4 0.0%
scsi_vhci`vhci_scsi_init_pkt 4 0.0%
genunix`mdi_select_path 4 0.0%
scsi`scsi_init_pkt 4 0.0%
zfs`rrw_enter 4 0.0%
zfs`arc_read 4 0.0%
unix`hment_mapcnt 4 0.0%
fcp`ssfcp_cmd_callback 4 0.0%
sd`xbuf_iostart 4 0.0%
unix`i_ddi_vaddr_rep_get8 4 0.0%
zfs`zio_root 4 0.0%
genunix`avl_find 4 0.0%
nvidia`_nv003011rm 4 0.0%
zfs`zio_notify_parent 4 0.0%
genunix`poll_common 4 0.0%
sd`sd_initpkt_for_buf 4 0.0%
unix`rw_write_held 4 0.0%
genunix`callout_execute 4 0.0%
zfs`vdev_mirror_child_done 4 0.0%
genunix`sleepq_unlink 4 0.0%
zfs`zio_null 4 0.0%
zfs`arc_do_user_evicts 4 0.0%
zfs`zio_interrupt 4 0.0%
sd`sd_start_cmds 4 0.0%
genunix`ioctl 5 0.0%
zfs`dnode_rele 5 0.0%
sd`sd_xbuf_init 5 0.0%
emlxs`emlxs_issue_iocb 5 0.0%
zfs`rrn_find_and_remove 5 0.0%
zfs`arc_evict_ghost 5 0.0%
zfs`zio_wait_for_children_done 5 0.0%
zfs`vdev_is_dead 5 0.0%
genunix`read32 5 0.0%
zfs`vdev_disk_io_intr 5 0.0%
zfs`arc_change_state 5 0.0%
zfs`zio_ready 5 0.0%
genunix`restore_mstate 5 0.0%
emlxs`emlxs_pcimem_bcopy 5 0.0%
genunix`sleepq_wakeall_chan 5 0.0%
genunix`mdi_pi_kstat_iosupdate 5 0.0%
sha1`sha1_block_data_order 5 0.0%
sd`sdstrategy 5 0.0%
zfs`vdev_mirror_io_done 5 0.0%
zfs`dmu_buf_rele_array 5 0.0%
zfs`vdev_mirror_io_start 5 0.0%
zfs`spa_get_failmode 5 0.0%
unix`htable_va2entry 5 0.0%
nvidia`_nv006149rm 5 0.0%
zfs`dbuf_hash_remove 5 0.0%
unix`sys_syscall 6 0.0%
unix`mutex_init 6 0.0%
unix`kstat_waitq_enter 6 0.0%
zfs`zio_wait 6 0.0%
unix`cmt_balance 6 0.0%
emlxs`emlxs_handle_ring_event 6 0.0%
scsi_vhci`vhci_intr 6 0.0%
zfs`vdev_queue_io_to_issue 6 0.0%
unix`bcopy 6 0.0%
sd`sd_return_command 6 0.0%
zfs`zio_checksum_verify 6 0.0%
zfs`arc_buf_add_ref 6 0.0%
rootnex`rootnex_coredma_freehdl 6 0.0%
emlxs`emlxs_send_fcp_cmd 6 0.0%
sd`sd_core_iostart 6 0.0%
unix`ddi_get16 6 0.0%
zfs`vdev_mirror_map_alloc 7 0.0%
emlxs`emlxs_pkt_to_bpl 7 0.0%
unix`tsc_scalehrtime 7 0.0%
sd`sdinfo 7 0.0%
TS`ts_wakeup 7 0.0%
genunix`sleepq_insert 7 0.0%
genunix`cpu_decay 7 0.0%
zfs`vdev_disk_io_start 7 0.0%
zfs`zio_vdev_io_done 7 0.0%
genunix`post_syscall 7 0.0%
genunix`timespectohz64 7 0.0%
unix`mutex_owned 7 0.0%
emlxs`emlxs_unregister_pkt 7 0.0%
genunix`cv_block 7 0.0%
unix`av_dispatch_autovect 7 0.0%
zfs`zfs_shim_read 8 0.0%
emlxs`emlxs_proc_ring 8 0.0%
zfs`arc_buf_alloc 8 0.0%
zfs`dbuf_create 8 0.0%
emlxs`emlxs_register_pkt 8 0.0%
zfs`dbuf_read_impl 8 0.0%
genunix`taskq_dispatch 8 0.0%
genunix`timeout_common 8 0.0%
unix`i_ddi_vaddr_rep_put8 8 0.0%
TS`ts_sleep 8 0.0%
zfs`zio_push_transform 9 0.0%
zfs`dbuf_whichblock 9 0.0%
genunix`getminor 9 0.0%
zfs`arc_buf_remove_ref 9 0.0%
zfs`buf_hash_remove 9 0.0%
zfs`arc_evict 9 0.0%
emlxs`emlxs_proc_attention 9 0.0%
genunix`fs_rwunlock 9 0.0%
unix`resume 9 0.0%
dtrace`dtrace_dynvar_clean 9 0.0%
emlxs`emlxs_mem_get 10 0.0%
emlxs`emlxs_pkt_init 10 0.0%
genunix`taskq_thread 10 0.0%
emlxs`emlxs_handle_fcp_event 10 0.0%
zfs`zio_checksum_error 10 0.0%
nvidia`_nv002899rm 10 0.0%
emlxs`emlxs_transport 10 0.0%
fcp`ssfcp_prepare_pkt 10 0.0%
genunix`ddi_dma_sync 10 0.0%
genunix`sleepq_wakeone_chan 10 0.0%
genunix`callout_schedule_1 10 0.0%
scsi_vhci`vhci_bind_transport 10 0.0%
unix`membar_enter 10 0.0%
unix`swtch 11 0.0%
rootnex`rootnex_coredma_allochdl 11 0.0%
zfs`dmu_buf_hold_array 11 0.0%
zfs`zfs_range_lock 11 0.0%
zfs`zfs_range_unlock 11 0.0%
unix`hat_switch 12 0.0%
zfs`dbuf_hold_impl 12 0.0%
genunix`avl_numnodes 12 0.0%
genunix`avl_remove 12 0.0%
unix`x86pte_access_pagetable 12 0.0%
unix`intr_thread 12 0.0%
pcplusmp`apic_send_ipi 12 0.0%
unix`idle 12 0.0%
nvidia`_nv003003rm 12 0.0%
unix`htable_getpage 12 0.0%
unix`htable_release 12 0.0%
unix`lwp_getdatamodel 13 0.0%
unix`setfrontdq 13 0.0%
genunix`cpu_update_pct 13 0.0%
zfs`zio_assess 13 0.0%
genunix`kmem_depot_alloc 13 0.0%
fcp`ssfcp_scsi_init_pkt 14 0.0%
nvidia`_nv003006rm 14 0.0%
genunix`fs_rwlock 14 0.0%
zfs`zio_wait_for_children 14 0.0%
zfs`dbuf_read 14 0.0%
genunix`kmem_free 14 0.0%
genunix`fop_read 14 0.0%
unix`lock_try 14 0.0%
unix`cpu_wakeup 14 0.0%
emlxs`emlxs_issue_iocb_cmd 14 0.0%
genunix`list_remove 15 0.0%
unix`bitset_in_set 15 0.0%
unix`x86pte_get 15 0.0%
rootnex`rootnex_get_sgl 15 0.0%
sd`sdintr 15 0.0%
zfs`dnode_hold_impl 15 0.0%
unix`htable_lookup 15 0.0%
unix`atomic_cas_32 15 0.0%
zfs`dmu_zfetch 16 0.0%
zfs`zio_done 17 0.0%
zfs`buf_hash_find 17 0.0%
unix`copyin 17 0.0%
genunix`disp_lock_exit_nopreempt 17 0.0%
zfs`dmu_read_uio 18 0.0%
unix`atomic_add_32 18 0.0%
zfs`dbuf_hash 18 0.0%
zfs`dmu_zfetch_find 18 0.0%
zfs`arc_released 18 0.0%
unix`htable_getpte 18 0.0%
unix`mutex_tryenter 18 0.0%
genunix`disp_lock_exit 19 0.0%
genunix`cv_broadcast 19 0.0%
zfs`dbuf_rele 20 0.0%
unix`hat_getpfnum 20 0.0%
unix`dosoftint 20 0.0%
genunix`nbl_need_check 21 0.0%
unix`lock_set 21 0.0%
zfs`arc_read_done 21 0.0%
genunix`gethrtime_unscaled 22 0.0%
unix`ddi_get32 22 0.0%
zfs`lzjb_decompress 24 0.0%
genunix`thread_lock 26 0.0%
zfs`zio_pop_transform 27 0.0%
unix`mutex_vector_exit 27 0.0%
mm`mmwrite 27 0.0%
unix`_resume_from_idle 27 0.0%
zfs`dmu_buf_hold_array_by_dnode 27 0.0%
genunix`watch_disable_addr 29 0.0%
unix`disp_anywork 29 0.0%
unix`page_next_scan_large 30 0.0%
zfs`zio_create 30 0.0%
zfs`dbuf_find 30 0.0%
genunix`kmem_cache_free 31 0.0%
genunix`kmem_cache_alloc 33 0.0%
zfs`zio_execute 33 0.0%
genunix`cdev_write 33 0.0%
genunix`fsflush 33 0.0%
zfs`zfs_read 33 0.0%
unix`disp 36 0.0%
genunix`read 37 0.0%
unix`_interrupt 38 0.0%
unix`bitset_atomic_add 39 0.0%
unix`rw_exit 39 0.0%
unix`mul32 39 0.0%
unix`pc_rtcget 40 0.0%
unix`clear_int_flag 41 0.0%
genunix`gethrestime 44 0.0%
genunix`clear_active_fd 44 0.0%
unix`setbackdq 48 0.0%
genunix`set_active_fd 48 0.0%
unix`bitset_atomic_del 48 0.0%
genunix`gethrtime 49 0.0%
genunix`fop_rwlock 51 0.0%
unix`gethrestime_sec 52 0.0%
zfs`buf_hash 53 0.0%
unix`rw_enter 60 0.0%
unix`lock_set_spl 61 0.0%
genunix`fop_rwunlock 62 0.0%
unix`atomic_add_64_nv 64 0.0%
genunix`write32 69 0.0%
mm`mmrw 72 0.0%
unix`atomic_add_64 74 0.0%
specfs`spec_maxoffset 75 0.0%
genunix`releasef 78 0.0%
unix`mutex_exit 83 0.0%
unix`pc_gethrestime 93 0.0%
nvidia`_nv002897rm 94 0.0%
genunix`fop_write 94 0.0%
genunix`copyin_nowatch 95 0.0%
genunix`getf 100 0.0%
specfs`smark 102 0.0%
unix`disp_getwork 112 0.0%
unix`bzero 113 0.0%
genunix`copyin_args32 124 0.0%
genunix`syscall_entry 149 0.1%
specfs`spec_write 163 0.1%
genunix`write 219 0.1%
unix`tsc_gethrtime_delta 278 0.1%
genunix`syscall_mstate 360 0.1%
unix`tsc_gethrtimeunscaled_delta 395 0.1%
unix`sys_syscall32 477 0.2%
genunix`fsflush_do_pages 582 0.2%
unix`mutex_enter 709 0.2%
unix`kcopy 1580 0.5%
zfs`fletcher_2_native 1965 0.7%
unix`cpu_halt 276440 95.9%
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
