I am testing attached volume storage on our openstack cluster which uses ceph for block storage. our Ceph nodes have large SSD's for their journals 50+GB for each OSD. I'm thinking some parameter is a little off because with relatively small writes I am seeing drastically reduced write speeds.
we have 2 nodes withs 12 total OSD's each with 50GB SSD Journal. here is our Ceph config [global] fsid = 19bc15fd-c0cc-4f35-acd2-292a86fbcf7d mon_initial_members = node-5 node-4 node-3 mon_host = 192.168.0.8 192.168.0.7 192.168.0.13 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true log_to_syslog_level = info log_to_syslog = True osd_pool_default_size = 1 osd_pool_default_min_size = 1 osd_pool_default_pg_num = 64 public_network = 192.168.0.0/24 log_to_syslog_facility = LOG_LOCAL0 osd_journal_size = 50000 auth_supported = cephx osd_pool_default_pgp_num = 64 osd_mkfs_type = xfs cluster_network = 192.168.1.0/24 osd_recovery_max_active = 1 osd_max_backfills = 1 [client] rbd_cache = True rbd_cache_writethrough_until_flush = True [client.radosgw.gateway] rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path = /tmp/radosgw.sock rgw_keystone_revocation_interval = 1000000 rgw_keystone_url = 192.168.0.2:35357 rgw_keystone_admin_token = ZBz37Vlv host = node-3 rgw_dns_name = *.ciminc.com rgw_print_continue = True rgw_keystone_token_cache_size = 10 rgw_data = /var/lib/ceph/radosgw user = www-data This is the degradation I am speaking of.. dd if=/dev/zero of=/mnt/ext4/output bs=1000k count=1k; rm -f /mnt/ext4/output; 1024+0 records in 1024+0 records out 1048576000 bytes (1.0 GB) copied, 0.887431 s, 1.2 GB/s dd if=/dev/zero of=/mnt/ext4/output bs=1000k count=2k; rm -f /mnt/ext4/output; 2048+0 records in 2048+0 records out 2097152000 bytes (2.1 GB) copied, 3.75782 s, 558 MB/s dd if=/dev/zero of=/mnt/ext4/output bs=1000k count=3k; rm -f /mnt/ext4/output; 3072+0 records in 3072+0 records out 3145728000 bytes (3.1 GB) copied, 10.0054 s, 314 MB/s dd if=/dev/zero of=/mnt/ext4/output bs=1000k count=5k; rm -f /mnt/ext4/output; 5120+0 records in 5120+0 records out 5242880000 bytes (5.2 GB) copied, 24.1971 s, 217 MB/s Any suggestions for improving the large write degradation?
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
