On Sat, Sep 27, 2014 at 8:14 AM, Timur Nurlygayanov <tnurlygaya...@mirantis.com> wrote: > Hello all, > > I installed OpenStack with Glance + Ceph OSD with replication factor 2 and > now I can see the write operations are extremly slow. > For example, I can see only 0.04 MB/s write speed when I run rados bench > with 512b blocks: > > rados bench -p test 60 write --no-cleanup -t 1 -b 512 > > Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0 > objects > Object prefix: benchmark_data_node-17.domain.tld_15862 > sec Cur ops started finished avg MB/s cur MB/s last lat > avg lat > 0 0 0 0 0 0 > - 0 > 1 1 83 82 0.0400341 0.0400391 > 0.008465 0.0120985 > 2 1 169 168 0.0410111 0.0419922 > 0.080433 0.0118995 > 3 1 240 239 0.0388959 0.034668 > 0.008052 0.0125385 > 4 1 356 355 0.0433309 0.0566406 > 0.00837 0.0112662 > 5 1 472 471 0.0459919 0.0566406 > 0.008343 0.0106034 > 6 1 550 549 0.0446735 0.0380859 > 0.036639 0.0108791 > 7 1 581 580 0.0404538 0.0151367 > 0.008614 0.0120654 > > > My test environment configuration: > Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per > node, HDDs WDC WD5003ABYX-01WERA0. > OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate > nodes). > CentOS 6.5, kernel 2.6.32-431.el6.x86_64. > > I tested several config options for optimizations, like in > /etc/ceph/ceph.conf: > > [default] > ... > osd_pool_default_pg_num = 1024 > osd_pool_default_pgp_num = 1024 > osd_pool_default_flag_hashpspool = true > ... > [osd] > osd recovery max active = 1 > osd max backfills = 1 > filestore max sync interval = 30 > filestore min sync interval = 29 > filestore flusher = false > filestore queue max ops = 10000 > filestore op threads = 16 > osd op threads = 16 > ... > [client] > rbd_cache = true > rbd_cache_writethrough_until_flush = true > > and in /etc/cinder/cinder.conf: > > [DEFAULT] > volume_tmp_dir=/tmp > > but in the result performance was increased only on ~30 % and it not looks > like huge success. > > Non-default mount options and TCP optimization increase the speed in about > 1%: > > [root@node-17 ~]# mount | grep ceph > /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs > (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0) > > [root@node-17 ~]# cat /etc/sysctl.conf > net.core.rmem_max = 16777216 > net.core.wmem_max = 16777216 > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 65536 16777216 > net.ipv4.tcp_window_scaling = 1 > net.ipv4.tcp_timestamps = 1 > net.ipv4.tcp_sack = 1 > > > Do we have other ways to significantly improve CEPH storage performance? > Any feedback and comments are welcome!
This is entirely latency dominated and OpenStack configuration changes aren't going to be able to do much — you're getting 80 sequential ops a second out of a system that has to do two round trips over a network and hit two hard drives on every operation. You might want to spend some time looking at how latency, bandwidth, and concurrency are (often not) related. :) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
