Thanks for your information. Here is result when I run atop on 1 Ceph HDD host: http://prntscr.com/iwmc86
There is some disk busy with over 100%, but the SSD journal (SSD) use only 3%, is it normal? Is there any way to optimize using of SSD journal? Could you give me some keyword? Here is configuration of Ceph HDD Host: Dell PowerEdge R730xd Server Quantity PE R730/xd Motherboard 1 Intel Xeon E5-2620 v4 2.1GHz,20M Cache,8.0GT/s QPI,Turbo,HT,8C/16T (85W) Max Mem 2133MHz 1 16GB RDIMM, 2400MT/s, Dual Rank, x8 Data Width 2 300GB 15K RPM SAS 12Gbps 2.5in Flex Bay Hard Drive - OS Drive (RAID 1) 2 4TB 7.2K RPM NLSAS 12Gbps 512n 3.5in Hot-plug Hard Drive - OSD Drive 7 200GB Solid State Drive SATA Mix Use MLC 6Gbps 2.5in Hot-plug Drive - Journal Drive (RAID 1) 2 PERC H730 Integrated RAID Controller, 1GB Cache *(we are using Writeback mode)* 1 Dual, Hot-plug, Redundant Power Supply (1+1), 750W 1 Broadcom 5720 QP 1Gb Network Daughter Card 1 QLogic 57810 Dual Port 10Gb Direct Attach/SFP+ Network Adapter 1 For some reasons, we can't configure Jumbo Frame in this cluster. We'll refer your suggest about scrub. 2018-03-26 7:41 GMT+07:00 Christian Balzer <ch...@gol.com>: > > Hello, > > in general and as reminder for others, the more information you supply, > the more likely are people to answer and answer with actually pertinent > information. > Since you haven't mentioned the hardware (actual HDD/SSD models, CPU/RAM, > controllers, etc) we're still missing a piece of the puzzle that could be > relevant. > > But given what we have some things are more likely than others. > Also, an inline 90KB screenshot of a TEXT iostat output is a bit of a > no-no, never mind that atop instead of top from the start would have given > you and us much more insight. > > On Sun, 25 Mar 2018 14:35:57 +0700 Sam Huracan wrote: > > > Thank you all. > > > > 1. Here is my ceph.conf file: > > https://pastebin.com/xpF2LUHs > > > As Lazlo noted (and it matches your iostat output beautifully), tuning > down scrubs is likely going to have an immediate beneficial impact, as > deep-scrubs in particular are VERY disruptive and I/O intense operations. > > However the "osd scrub sleep = 0.1" may make things worse in certain Jewel > versions, as they all went through the unified queue and this would cause > a sleep for ALL operations, not just the scrub ones. > I can't remember when this was fixed and the changelog is of no help, so > hopefully somebody who knows will pipe up. > If in doubt of course, experiment. > > In addition to that, if you have low usage times, set > your osd_scrub_(start|end)_hour accordingly and also check the ML archives > for other scrub scheduling tips. > > I'd also leave these: > filestore max sync interval = 100 > filestore min sync interval = 50 > filestore queue max ops = 5000 > filestore queue committing max ops = 5000 > journal max write entries = 1000 > journal queue max ops = 5000 > > at their defaults, playing with those parameters requires a good > understanding of how Ceph filestore works AND usually only makes sense > with SSD/NVMe setups. > Especially the first 2 could lead to quite the IO pileup. > > > > 2. Here is result from ceph -s: > > root@ceph1:/etc/ceph# ceph -s > > cluster 31154d30-b0d3-4411-9178-0bbe367a5578 > > health HEALTH_OK > > monmap e3: 3 mons at {ceph1= > > 10.0.30.51:6789/0,ceph2=10.0.30.52:6789/0,ceph3=10.0.30.53:6789/0} > > election epoch 18, quorum 0,1,2 ceph1,ceph2,ceph3 > > osdmap e2473: 63 osds: 63 up, 63 in > > flags sortbitwise,require_jewel_osds > > pgmap v34069952: 4096 pgs, 6 pools, 21534 GB data, 5696 kobjects > > 59762 GB used, 135 TB / 194 TB avail > > 4092 active+clean > > 2 active+clean+scrubbing > > 2 active+clean+scrubbing+deep > > client io 36096 kB/s rd, 41611 kB/s wr, 1643 op/s rd, 1634 op/s wr > > > See above about deep-scrub, which will read ALL the objects of the PG > being scrubbed and thus not only saturates the OSDs involved with reads > but ALSO dirties the pagecache with cold objects, making other reads on > the nodes slow by requiring them to hit the disks, too. > > It would be interesting to see a "ceph -s" when your cluster is busy but > NOT scrubbing, 1600 write op/s are about what 21 HDDs can handle. > So for the time being, disable scrubs entirely and see if your problems > go away. > If so, you now know the limits of your current setup and will want to > avoid hitting them again. > > Having a dedicated SSD pool for high-end VMs or a cache-tier (if it is a > fit, not likely in your case) would be a way forward if your client > demands are still growing. > > Christian > > > > > > > 3. We use 1 SSD for journaling 7 HDD (/dev/sdi), I set 16GB for each > > journal, here is result from ceph-disk list command: > > > > /dev/sda : > > /dev/sda1 ceph data, active, cluster ceph, osd.0, journal /dev/sdi1 > > /dev/sdb : > > /dev/sdb1 ceph data, active, cluster ceph, osd.1, journal /dev/sdi2 > > /dev/sdc : > > /dev/sdc1 ceph data, active, cluster ceph, osd.2, journal /dev/sdi3 > > /dev/sdd : > > /dev/sdd1 ceph data, active, cluster ceph, osd.3, journal /dev/sdi4 > > /dev/sde : > > /dev/sde1 ceph data, active, cluster ceph, osd.4, journal /dev/sdi5 > > /dev/sdf : > > /dev/sdf1 ceph data, active, cluster ceph, osd.5, journal /dev/sdi6 > > /dev/sdg : > > /dev/sdg1 ceph data, active, cluster ceph, osd.6, journal /dev/sdi7 > > /dev/sdh : > > /dev/sdh3 other, LVM2_member > > /dev/sdh1 other, vfat, mounted on /boot/efi > > /dev/sdi : > > /dev/sdi1 ceph journal, for /dev/sda1 > > /dev/sdi2 ceph journal, for /dev/sdb1 > > /dev/sdi3 ceph journal, for /dev/sdc1 > > /dev/sdi4 ceph journal, for /dev/sdd1 > > /dev/sdi5 ceph journal, for /dev/sde1 > > /dev/sdi6 ceph journal, for /dev/sdf1 > > /dev/sdi7 ceph journal, for /dev/sdg1 > > > > 4. With iostat, we just run "iostat -x 2", /dev/sdi is journal SSD, > > /dev/sdh is OS Disk, and the rest is OSD Disks. > > root@ceph1:/etc/ceph# lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sda 8:0 0 3.7T 0 disk > > └─sda1 8:1 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-0 > > sdb 8:16 0 3.7T 0 disk > > └─sdb1 8:17 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-1 > > sdc 8:32 0 3.7T 0 disk > > └─sdc1 8:33 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-2 > > sdd 8:48 0 3.7T 0 disk > > └─sdd1 8:49 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-3 > > sde 8:64 0 3.7T 0 disk > > └─sde1 8:65 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-4 > > sdf 8:80 0 3.7T 0 disk > > └─sdf1 8:81 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-5 > > sdg 8:96 0 3.7T 0 disk > > └─sdg1 8:97 0 3.7T 0 part > > /var/lib/ceph/osd/ceph-6 > > sdh 8:112 0 278.9G 0 disk > > ├─sdh1 8:113 0 512M 0 part /boot/efi > > └─sdh3 8:115 0 278.1G 0 part > > ├─hnceph--hdd1--vg-swap (dm-0) 252:0 0 59.6G 0 lvm [SWAP] > > └─hnceph--hdd1--vg-root (dm-1) 252:1 0 218.5G 0 lvm / > > sdi 8:128 0 185.8G 0 disk > > ├─sdi1 8:129 0 16.6G 0 part > > ├─sdi2 8:130 0 16.6G 0 part > > ├─sdi3 8:131 0 16.6G 0 part > > ├─sdi4 8:132 0 16.6G 0 part > > ├─sdi5 8:133 0 16.6G 0 part > > ├─sdi6 8:134 0 16.6G 0 part > > └─sdi7 8:135 0 16.6G 0 part > > > > Could you give me some idea to continue check? > > > > > > 2018-03-25 12:25 GMT+07:00 Budai Laszlo <laszlo.bu...@gmail.com>: > > > > > could you post the result of "ceph -s" ? besides the health status > there > > > are other details that could help, like the status of your PGs., also > the > > > result of "ceph-disk list" would be useful to understand how your > disks are > > > organized. For instance with 1 SSD for 7 HDD the SSD could be the > > > bottleneck. > > > From the outputs you gave us we don't know which are the spinning disks > > > and which is the ssd (looking at the numbers I suspect that sdi is your > > > SSD). we also don't kow what parameters were you using when you've ran > the > > > iostat command. > > > > > > Unfortunately it's difficult to help you without knowing more about > your > > > system. > > > > > > Kind regards, > > > Laszlo > > > > > > On 24.03.2018 20:19, Sam Huracan wrote: > > > > This is from iostat: > > > > > > > > I'm using Ceph jewel, has no HW error. > > > > Ceph health OK, we've just use 50% total volume. > > > > > > > > > > > > 2018-03-24 22:20 GMT+07:00 <c...@elchaka.de <mailto:c...@elchaka.de > >>: > > > > > > > > I would Check with Tools like atop the utilization of your Disks > > > also. Perhaps something Related in dmesg or dorthin? > > > > > > > > - Mehmet > > > > > > > > Am 24. März 2018 08:17:44 MEZ schrieb Sam Huracan < > > > nowitzki.sa...@gmail.com <mailto:nowitzki.sa...@gmail.com>>: > > > > > > > > > > > > Hi guys, > > > > We are running a production OpenStack backend by Ceph. > > > > > > > > At present, we are meeting an issue relating to high iowait > in > > > VM, in some MySQL VM, we see sometime IOwait reaches abnormal high > peaks > > > which lead to slow queries increase, despite load is stable (we test > with > > > script simulate real load), you can see in graph. > > > > https://prnt.sc/ivndni > > > > > > > > MySQL VM are place on Ceph HDD Cluster, with 1 SSD journal > for 7 > > > HDD. In this cluster, IOwait on each ceph host is about 20%. > > > > https://prnt.sc/ivne08 > > > > > > > > > > > > Can you guy help me find the root cause of this issue, and > how > > > to eliminate this high iowait? > > > > > > > > Thanks in advance. > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com < > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Rakuten Communications >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
