so, anyone have any ideas? I'm obviously hitting a bug here. I'm happy to help anyone solve this, I DESPERATELY need this data. I can post dtrace results if you send them to me. I wish I could solve this myself, but I'm not a C progammer, I don't know how to program filesystems, much less an advanced filesystem as ZFS.
Also, this is not a weird configuration, or alpha software, or a hack I'm
making. This is a "production" server with proven software, on which I executed
a regular maintenance operation, just a clone destroy, which should take just a
few seconds, and I found so many wrong things:
* a clone destroy takes more than a few seconds or minutes
* to destroy the clone it tries to read, apparently, through the whole dataset
* while reading through the whole dataset, it does so at 1MB/s (dataset size is
450GB without snapshots)
* it loads these reads to the ARC, filling it up and freezing the machine
* the ARC doesn't work in this case as all reads have 100% miss ratio
* it doesn't respect the ARC maximum size.
* after a reboot, it won't work on the background. it will keep the pool locked
preventing mounts
* disabling the ZIL doesn't prevent zpool from trying to complete the destroy
this is very very wrong.
I don't know anything about dtrace. I wrote this, I don't really know if this
is correct. I hope it's of use to someone.
#!/usr/sbin/dtrace -s
fbt:zfs::
{
@count_table[probefunc]=count();
}
tick-300s {exit(0);}
The output for the 5 minute run:
dnode_buf_pageout 6
zfs_sync 12
dmu_zfetch_rele 18
dnode_destroy 18
bplist_close 78
bplist_open 78
bplist_sync 78
dbuf_hold 78
dmu_buf_get_user 78
dmu_object_info 78
dmu_object_info_from_dnode 78
dmu_zfetch 78
dmu_zfetch_find 78
dnode_hold 78
dnode_hold_impl 78
dsl_pool_sync 78
dsl_pool_zil_clean 78
spa_async_dispatch 78
spa_errlog_sync 78
spa_scrub_resume 78
spa_scrub_suspend 78
spa_sync 78
spa_sync_config_object 78
txg_quiesce 78
uberblock_update 78
vdev_config_sync 78
dmu_tx_commit 156
dmu_tx_create_assigned 156
dmu_tx_create_dd 156
spa_sync_aux_dev 156
txg_list_empty 234
arc_do_user_evicts 368
txg_list_remove 390
txg_thread_wait 390
add_reference 486
arc_buf_add_ref 486
spa_config_enter 566
spa_config_exit 566
zio_null 566
zio_root 566
zio_wait 566
vdev_queue_agg_io_done 1954
dbuf_hold_level 62208
dmu_tx_check_ioerr 62208
dnode_next_offset 62208
dnode_next_offset_level 62208
dbuf_whichblock 62286
dbuf_do_evict 62436
dbuf_hash_remove 62436
dbuf_clear 62440
dbuf_destroy 62440
dbuf_evict 62440
dbuf_evict_user 62440
arc_space_return 62462
arc_cksum_compute 62510
arc_read_done 62510
arc_set_callback 62510
dbuf_read_done 62510
dbuf_set_data 62510
fletcher_4_native 62510
lzjb_decompress 62510
vdev_mirror_child_done 62510
vdev_mirror_io_done 62510
vdev_mirror_map_free 62510
vdev_raidz_io_done 62510
vdev_raidz_map_free 62510
zio_checksum_error 62510
zio_checksum_verified 62510
zio_decompress_data 62510
zio_read_decompress 62510
dnode_rele 62514
arc_buf_destroy 62694
arc_cksum_verify 62694
arc_evict 62694
arc_adapt 62696
arc_buf_alloc 62696
arc_evict_needed 62696
arc_get_data_buf 62696
arc_read 62696
buf_hash_insert 62696
dbuf_create 62696
dbuf_findbp 62696
dbuf_hash_insert 62696
dbuf_read_impl 62696
hdr_cons 62696
spa_get_failmode 62696
vdev_mirror_child_select 62696
vdev_mirror_io_start 62696
vdev_mirror_map_alloc 62696
vdev_raidz_io_start 62696
vdev_raidz_map_alloc 62696
zio_read 62696
zio_read_init 62696
arc_space_consume 62698
spa_get_random 62774
arc_access 62996
arc_buf_freeze 63002
arc_buf_remove_ref 63002
remove_reference 63002
arc_reclaim_needed 63080
zio_ready 63262
zio_wait_for_children_ready 63262
arc_change_state 125204
buf_hash_find 125206
vdev_lookup_top 125392
dbuf_add_ref 125470
arc_released 125512
vdev_cache_fill 153525
vdev_cache_evict 154090
zio_vdev_io_reissue 155192
vdev_disk_io_intr 163696
zio_interrupt 163696
vdev_disk_io_done 163708
vdev_queue_io_done 163708
vdev_error_inject 163924
vdev_queue_io_remove 166100
vdev_cache_allocate 166298
vdev_queue_io_add 166434
vdev_cache_hit 175179
zio_vdev_io_bypass 179874
dbuf_find 187190
dbuf_hold_impl 187190
dbuf_read 187190
vdev_raidz_child_done 187536
dbuf_update_data 249700
zio_buf_free 281068
zio_buf_alloc 281438
zio_vdev_io_done 288728
dbuf_rele 312214
dbuf_hash 312322
vdev_dtl_contains 313480
buf_hash 314084
vdev_queue_io 323580
vdev_stat_update 405526
zio_vdev_child_io 406828
vdev_readable 414708
vdev_queue_io_to_issue 437894
zio_should_retry 468036
zio_vdev_io_assess 468036
zio_done 468601
zio_assess 468602
zio_clear_transform_stack 468602
zio_nowait 469524
zio_create 470090
vdev_is_dead 477404
vdev_cache_read 499324
vdev_disk_io_start 499324
zio_notify_parent 530730
zio_pop_transform 531112
zio_push_transform 532786
zio_wait_for_children_done 593994
spa_state 624716
zio_vdev_io_start 624716
zio_wait_for_children 657256
vdev_queue_deadline_compare 906304
zio_execute 1247615
vdev_queue_offset_compare 1694848
vdev_cache_lastused_compare 2472113
vdev_cache_offset_compare 3076216
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
