Hi Chris, The osds are running on arm nodes. Every node has a two core 1.5Ghz arm 32bit cpu and 2G ram and runs 2 osds.Hdd is 10TB and journal colocate with data on the same disk. Drives are half full now,but the problem I described also happened when the hdds are empty. Filesystem is ext4 because I have some problem to run xfs for now. I am trying to better balance the pg distrubition now to see if it can ease the high latency problem.
2018-03-10 shadowlin 发件人：Chris Hoy Poy <chr...@base3.com.au> 发送时间：2018-03-10 09:44 主题：Re: [ceph-users] [jewel] High fs_apply_latency osds 收件人："shadow_lin"<shadow_...@163.com>,"ceph-users"<ceph-users@lists.ceph.com> 抄送： Hi Shadowlin, Can you describe your hardware ? Cpu/ram/harddrives involved etc Also how your drives are set up? How full are the drives? What filesystem is it? Cheers /chris Sent from my SAMSUNG Galaxy S6 on the Telstra Mobile Network -------- Original message -------- From: shadow_lin <shadow_...@163.com> Date: 10/3/18 1:41 am (GMT+08:00) To: ceph-users <ceph-users@lists.ceph.com> Subject: [ceph-users] [jewel] High fs_apply_latency osds Hi list, During my write test,I find there are always some of the osds have high fs_apply_latency（1k-5kms,2-8times more than others）. At first I think it is caused by unbalanced pg distribution, but after I reweight the osds the problem hasn't gone away. I looked into the osds with high latency and find one thing in common.There are alot of read activities on these osds.Result of iostat shows there are 400-500 r/s and 2000-3000 rKB/s on these high latnecy osds while the normal osds have around 100 r/s 300-400 rKB/s. I tried to restart osd daemon with high latency.It did became normal sometime but there will be another high latency osds come out.And the new high latency osd has the same high read activities. What are these read activities for? Is there a way to lower the latency? Thanks 2018-03-10 shadowlin Message protected by MailGuard: e-mail anti-virus, anti-spam and content filtering. http://www.mailguard.com.au Report this message as spam Message protected by MailGuard: e-mail anti-virus, anti-spam and content filtering. http://www.mailguard.com.au
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
