I compare the applied commits between 0.80.7 and 0.80.8 , then I focus on this： https://github.com/ceph/ceph/commit/711a7e6f81983ff2091caa0f232af914a04a041c?diff=unified#diff-9bcd2f7647a2bd574b6ebe6baf8e61b3
this commits seems take waitfor_read flushed out of the while cycle, maybe this cause the read request performance become worse? sorry if I was wrong , I am not familiar with the osdc source code . ， m 2015-02-21 23:14 GMT+08:00 Alexandre DERUMIER <aderum...@odiso.com>: > >>so maybe can sure this problem is cause from 0.80.8 > > That's a good news if you are sure that it's come from 0.80.8 > > The applied commits between 0.80.7 and 0.80.8 is here: > > https://github.com/ceph/ceph/compare/v0.80.7...v0.80.8 > > > Now, we need to find which of them is for librados/librbd. > > > ----- Mail original ----- > De: "杨万元" <yangwanyuan8...@gmail.com> > À: "aderumier" <aderum...@odiso.com> > Cc: "ceph-users" <ceph-users@lists.ceph.com> > Envoyé: Vendredi 13 Février 2015 04:39:05 > Objet: Re: [ceph-users] re： Upgrade 0.80.5 to 0.80.8 --the VM's read > requestbecome too slow > > thanks very much for your advice . > yes，as you said，disabled the rbd_cache will improve the read request，but > if i disabled rbd_cache， the randwrite request will be worse. so this > method maybe can not solve my problem, is it ? > In addition , I also test the 0.80.6 and 0.80.7 librbd,they are as good as > 0.80.5 performance , so maybe can sure this problem is cause from 0.80.8 > > 2015-02-12 19:33 GMT+08:00 Alexandre DERUMIER < aderum...@odiso.com > : > > > >>Hi, > >>Can you test with disabling rbd_cache ? > > >>I remember of a bug detected in giant, not sure it's also the case for > fireflt > > This was this tracker: > > http://tracker.ceph.com/issues/9513 > > But It has been solved and backported to firefly. > > Also, can you test 0.80.6 and 0.80.7 ? > > > > > > > > ----- Mail original ----- > De: "killingwolf" < killingw...@qq.com > > À: "ceph-users" < ceph-users@lists.ceph.com > > Envoyé: Jeudi 12 Février 2015 12:16:32 > Objet: [ceph-users] re： Upgrade 0.80.5 to 0.80.8 --the VM's read > requestbecome too slow > > I have this problems too , Help! > > ------------------ 原始邮件 ------------------ > 发件人: "杨万元";< yangwanyuan8...@gmail.com >; > 发送时间: 2015年2月12日(星期四) 中午11:14 > 收件人: " ceph-users@lists.ceph.com "< ceph-users@lists.ceph.com >; > 主题: [ceph-users] Upgrade 0.80.5 to 0.80.8 --the VM's read requestbecome > too slow > > Hello! > We use Ceph+Openstack in our private cloud. Recently we upgrade our > centos6.5 based cluster from Ceph Emperor to Ceph Firefly. > At first,we use redhat yum repo epel to upgrade, this Ceph's version is > 0.80.5. First upgrade monitor,then osd,last client. when we complete this > upgrade, we boot a VM on the cluster,then use fio to test the io > performance. The io performance is as better as before. Everything is ok! > Then we upgrade the cluster from 0.80.5 to 0.80.8,when we completed , we > reboot the VM to load the newest librbd. after that we also use fio to test > the io performance .then we find the randwrite and write is as good as > before.but the randread and read is become worse, randwrite's iops from > 4000-5000 to 300-400 ,and the latency is worse. the write's bw from 400MB/s > to 115MB/s . then I downgrade the ceph client version from 0.80.8 to > 0.80.5, then the reslut become normal. > So I think maybe something cause about librbd. I compare the 0.80.8 > release notes with 0.80.5 ( > http://ceph.com/docs/master/release-notes/#v0-80-8-firefly ), I just find > this change in 0.80.8 is something about read request : librbd: cap memory > utilization for read requests (Jason Dillaman) . Who can explain this？ > > > My ceph cluster is 400osd,5mons : > ceph -s > health HEALTH_OK > monmap e11: 5 mons at {BJ-M1-Cloud71= > 172.28.2.71:6789/0,BJ-M1-Cloud73=172.28.2.73:6789/0,BJ-M2-Cloud80=172.28.2.80:6789/0,BJ-M2-Cloud81=172.28.2.81:6789/0,BJ-M3-Cloud85=172.28.2.85:6789/0 > }, election epoch 198, quorum 0,1,2,3,4 > BJ-M1-Cloud71,BJ-M1-Cloud73,BJ-M2-Cloud80,BJ-M2-Cloud81,BJ-M3-Cloud85 > osdmap e120157: 400 osds: 400 up, 400 in > pgmap v26161895: 29288 pgs, 6 pools, 20862 GB data, 3014 kobjects > 41084 GB used, 323 TB / 363 TB avail > 29288 active+clean > client io 52640 kB/s rd, 32419 kB/s wr, 5193 op/s > > > The follwing is my ceph client conf : > [global] > auth_service_required = cephx > filestore_xattr_use_omap = true > auth_client_required = cephx > auth_cluster_required = cephx > mon_host = > 172.29.204.24,172.29.204.48,172.29.204.55,172.29.204.58,172.29.204.73 > mon_initial_members = ZR-F5-Cloud24, ZR-F6-Cloud48, ZR-F7-Cloud55, > ZR-F8-Cloud58, ZR-F9-Cloud73 > fsid = c01c8e28-304e-47a4-b876-cb93acc2e980 > mon osd full ratio = .85 > mon osd nearfull ratio = .75 > public network = 172.29.204.0/24 > mon warn on legacy crush tunables = false > > [osd] > osd op threads = 12 > filestore journal writeahead = true > filestore merge threshold = 40 > filestore split multiple = 8 > > [client] > rbd cache = true > rbd cache writethrough until flush = false > rbd cache size = 67108864 > rbd cache max dirty = 50331648 > rbd cache target dirty = 33554432 > > [client.cinder] > admin socket = /var/run/ceph/rbd-$pid.asok > > > > My VM is 8core16G,we use fio scripts is : > fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randread -size=60G > -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 > fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=randwrite -size=60G > -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 > fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=read -size=60G > -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 > fio -ioengine=libaio -bs=4k -direct=1 -thread -rw=write -size=60G > -filename=/dev/vdb -name="EBS" -iodepth=32 -runtime=200 > > The following is the io test result > ceph client verison :0.80.5 > read: bw= 430MB > write: bw=420MB > randread: iops= 4875 latency=65ms > randwrite: iops=6844 latency=46ms > > ceph client verison :0.80.8 > read: bw= 115MB > write: bw=480MB > randread: iops= 381 latency=83ms > randwrite: iops=4843 latency=68ms > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
