> Still don't know your Ceph version, is it the latest Jewel?
10.1.2-0ubuntu1
> Check the ML archives, I remember people having performance issues with the
4.4 kernels.
Yes, I try today to find something
> These are OSD parameters, you need to query an OSD daemon.
There are:
ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show|grep
"filestore_queue"
"filestore_queue_max_ops": "3000",
"filestore_queue_max_bytes": "1048576000",
"filestore_queue_max_delay_multiple": "0",
"filestore_queue_high_delay_multiple": "0",
"filestore_queue_low_threshhold": "0.3",
"filestore_queue_high_threshhold": "0.9",
> Scrubbing is a major performance killer, especially on non-SSD journal
> OSDs and with older Ceph versions and/or non-tuned parameters:
I can't change those parametres on fly:
ceph tell osd.* injectargs '--osd_scrub_end_hour=6'
osd.0: osd_scrub_end_hour = '6' (unchangeable)
osd.1: osd_scrub_end_hour = '6' (unchangeable)
osd.2: osd_scrub_end_hour = '6' (unchangeable)
...
I try to change it today sone later and restart OSDs.
>Понедельник, 11 июля 2016, 12:38 +05:00 от Christian Balzer <ch...@gol.com>:
>
>
>Hello,
>
>On Mon, 11 Jul 2016 09:54:59 +0300 K K wrote:
>
>>
>> > I hope the fastest of these MONs (CPU and storage) has the lowest IP
>> > number and thus is the leader.
>> no, the lowest IP has slowest CPU. But zabbix didn't show any load at all
>> mons.
>
>In your use case and configuration no surprise, but again, the lowest IP
>will be leader by default and thus the busiest.
>
>> > Also what Ceph, OS, kernel version?
>>
>> ubuntu 16.04 kernel 4.4.0-22
>>
>Check the ML archives, I remember people having performance issues with the
>4.4 kernels.
>
>Still don't know your Ceph version, is it the latest Jewel?
>
>> > Two GbE ports, given the "frontend" up there with the MON description I
>> > assume that's 1 port per client (front) and cluster (back) network?
>> yes, one GbE for ceph client, one GbE for back network.
>OK, so (from a single GbE client) 100MB/s at most.
>
>> > Is there any other client on than that Windows VM on your Ceph cluster?
>> Yes, another one instance but without load.
>OK.
>
>> > Is Ceph understanding this now?
>> > Other than that, the queue options aren't likely to do much good with pure
>> >HDD OSDs.
>>
>> I can't find those parameter in running config:
>> ceph --admin-daemon /var/run/ceph/ceph-mon.block01.asok config show|grep
>> "filestore_queue"
>
>These are OSD parameters, you need to query an OSD daemon.
>
>> "filestore_queue_max_ops": "3000",
>> "filestore_queue_max_bytes": "1048576000",
>> "filestore_queue_max_delay_multiple": "0",
>> "filestore_queue_high_delay_multiple": "0",
>> "filestore_queue_low_threshhold": "0.3",
>> "filestore_queue_high_threshhold": "0.9",
>> > That should be 512, 1024 really with one RBD pool.
>>
>> Yes, I know. Today for test I added scbench pool with 128 pg
>> There are output status and osd tree:
>> ceph status
>> cluster 830beb43-9898-4fa9-98c1-ee04c1cdf69c
>> health HEALTH_OK
>> monmap e6: 3 mons at
>> {block01=10.30.9.21:6789/0,object01=10.30.9.129:6789/0,object02=10.30.9.130:6789/0}
>> election epoch 238, quorum 0,1,2 block01,object01,object02
>> osdmap e6887: 18 osds: 18 up, 18 in
>> pgmap v9738812: 1280 pgs, 3 pools, 17440 GB data, 4346 kobjects
>> 35049 GB used, 15218 GB / 50267 GB avail
>> 1275 active+clean
>> 3 active+clean+scrubbing+deep
>> 2 active+clean+scrubbing
>>
>Check the ML archives and restrict scrubs to off-peak hours as well as
>tune things to keep their impact low.
>
>Scrubbing is a major performance killer, especially on non-SSD journal
>OSDs and with older Ceph versions and/or non-tuned parameters:
>---
>osd_scrub_end_hour = 6
>osd_scrub_load_threshold = 2.5
>osd_scrub_sleep = 0.1
>---
>
>> client io 5030 kB/s rd, 1699 B/s wr, 19 op/s rd, 0 op/s wr
>>
>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> -1 54.00000 root default
>> -2 27.00000 host cn802
>> 0 3.00000 osd.0 up 1.00000 1.00000
>> 2 3.00000 osd.2 up 1.00000 1.00000
>> 4 3.00000 osd.4 up 1.00000 1.00000
>> 6 3.00000 osd.6 up 0.89995 1.00000
>> 8 3.00000 osd.8 up 1.00000 1.00000
>> 10 3.00000 osd.10 up 1.00000 1.00000
>> 12 3.00000 osd.12 up 0.89999 1.00000
>> 16 3.00000 osd.16 up 1.00000 1.00000
>> 18 3.00000 osd.18 up 0.90002 1.00000
>> -3 27.00000 host cn803
>> 1 3.00000 osd.1 up 1.00000 1.00000
>> 3 3.00000 osd.3 up 0.95316 1.00000
>> 5 3.00000 osd.5 up 1.00000 1.00000
>> 7 3.00000 osd.7 up 1.00000 1.00000
>> 9 3.00000 osd.9 up 1.00000 1.00000
>> 11 3.00000 osd.11 up 0.95001 1.00000
>> 13 3.00000 osd.13 up 1.00000 1.00000
>> 17 3.00000 osd.17 up 0.84999 1.00000
>> 19 3.00000 osd.19 up 1.00000 1.00000
>> > Wrong way to test this, test it from a monitor node, another client node
>> > (like your openstack nodes).
>> > In your 2 node cluster half of the reads or writes will be local, very
>> > much skewing your results.
>> I have been tested from copmute node also and have same result. 80-100Mb/sec
>>
>That's about as good as it gets (not 148MB/s, though!).
>But rados bench is not the same as real client I/O.
>
>> > Very high max latency, telling us that your cluster ran out of steam at
>> some point.
>>
>> I copying data from my windows instance right now.
>
>Re-do any testing when you've stopped all scrubbing.
>
>> > I'd de-frag anyway, just to rule that out.
>>
>>
>> >When doing your tests or normal (busy) operations from the client VM, run
>> > atop on your storage nodes and observe your OSD HDDs.
>> > Do they get busy, around 100%?
>>
>> Yes, high IO load (600-800 io). But this is very strange on SATA HDD. All
>> HDD have own OSD daemon and presented in OS as hardware RAID0(each block
>> node have hardware RAID). Example:
>
>Your RAID controller and its HW cache are likely to help with that speed,
>also all of these are reads, most likely the scrubs above, not a single
>write to be seen.
>
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 1.44 0.00 3.56 17.56 0.00 77.44
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await
>> w_await svctm %util
>> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdc 0.00 0.00 649.00 0.00 82912.00 0.00 255.51 8.30 12.74 12.74 0.00 1.26
>> 81.60
>> sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdf 0.00 0.00 761.00 0.00 94308.00 0.00 247.85 8.66 11.26 11.26 0.00 1.18
>> 90.00
>> sdg 0.00 0.00 761.00 0.00 97408.00 0.00 256.00 7.80 10.22 10.22 0.00 1.01
>> 76.80
>> sdh 0.00 0.00 801.00 0.00 102344.00 0.00 255.54 8.05 10.05 10.05 0.00 0.96
>> 76.80
>> sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> sdj 0.00 0.00 537.00 0.00 68736.00 0.00 256.00 5.54 10.26 10.26 0.00 0.98
>> 52.80
>>
>>
>> > Check with iperf or NPtcp that your network to the clients from the
>> > storage nodes is fully functional.
>> The network have been tested by iperf. 950-970Mbit among all nodes in
>> clustes (openstack and ceph)
>
>Didn't think it was that, one thing off the list to check.
>
>Christian
>
>Понедельник, 11 июля 2016, 10:58 +05:00 от Christian Balzer
>< ch...@gol.com >:
>> >
>> >
>> >Hello,
>> >
>> >On Mon, 11 Jul 2016 07:35:02 +0300 K K wrote:
>> >
>> >>
>> >> Hello, guys
>> >>
>> >> I to face a task poor performance into windows 2k12r2 instance running
>> >> on rbd (openstack cluster). RBD disk have a size 17Tb. My ceph cluster
>> >> consist from:
>> >> - 3 monitors nodes (Celeron G530/6Gb RAM, DualCore E6500/2Gb RAM,
>> >> Core2Duo E7500/2Gb RAM). Each node have 1Gbit network to frontend subnet
>> >> od Ceph cluster
>> >
>> >I hope the fastest of these MONs (CPU and storage) has the lowest IP
>> >number and thus is the leader.
>> >
>> >Also what Ceph, OS, kernel version?
>> >
>> >> - 2 block nodes (Xeon E5620/32Gb RAM/2*1Gbit NIC). Each node have
>> >> 2*500Gb HDD for operation system and 9*3Tb SATA HDD (WD SE). Total 18
>> >> OSD daemons on 2 nodes.
>> >
>> >Two GbE ports, given the "frontend" up there with the MON description I
>> >assume that's 1 port per client (front) and cluster (back) network?
>> >
>> >>Journals placed on same HDD as a rados data. I
>> >> know that better using for those purpose separate SSD disk.
>> >Indeed...
>> >
>> >>When I test
>> >> new windows instance performance was good (read/write something about
>> >> 100Mb/sec). But after I copied 16Tb data to windows instance read
>> >> performance has down to 10Mb/sec. Type of data on VM - image and video.
>> >>
>> >100MB/s would be absolute perfect with the setup you have, assuming no
>> >contention (other clients).
>> >
>> >Is there any other client on than that Windows VM on your Ceph cluster?
>> >
>> >> ceph.conf on client side:
>> >> [global]
>> >> auth cluster required = cephx
>> >> auth service required = cephx
>> >> auth client required = cephx
>> >> filestore xattr use omap = true
>> >> filestore max sync interval = 10
>> >> filestore queue max ops = 3000
>> >> filestore queue commiting max bytes = 1048576000
>> >> filestore queue commiting max ops = 5000
>> >> filestore queue max bytes = 1048576000
>> >> filestore queue committing max ops = 4096
>> >> filestore queue committing max bytes = 16 MiB
>> > ^^^
>> >Is Ceph understanding this now?
>> >Other than that, the queue options aren't likely to do much good with pure
>> >HDD OSDs.
>> >
>> >> filestore op threads = 20
>> >> filestore flusher = false
>> >> filestore journal parallel = false
>> >> filestore journal writeahead = true
>> >> journal dio = true
>> >> journal aio = true
>> >> journal force aio = true
>> >> journal block align = true
>> >> journal max write bytes = 1048576000
>> >> journal_discard = true
>> >> osd pool default size = 2 # Write an object n times.
>> >> osd pool default min size = 1
>> >> osd pool default pg num = 333
>> >> osd pool default pgp num = 333
>> >That should be 512, 1024 really with one RBD pool.
>> > http://ceph.com/pgcalc/
>> >
>> >> osd crush chooseleaf type = 1
>> >>
>> >> [client]
>> >> rbd cache = true
>> >> rbd cache size = 67108864
>> >> rbd cache max dirty = 50331648
>> >> rbd cache target dirty = 33554432
>> >> rbd cache max dirty age = 2
>> >> rbd cache writethrough until flush = true
>> >>
>> >>
>> >> rados bench show from block node show:
>> >Wrong way to test this, test it from a monitor node, another client node
>> >(like your openstack nodes).
>> >In your 2 node cluster half of the reads or writes will be local, very
>> >much skewing your results.
>> >
>> >> rados bench -p scbench 120 write --no-cleanup
>> >
>> >Default tests with 4MB "blocks", what are the writes or reads from you
>> >client VM like?
>> >
>> >> Total time run: 120.399337
>> >> Total writes made: 3538
>> >> Write size: 4194304
>> >> Object size: 4194304
>> >> Bandwidth (MB/sec): 117.542
>> >> Stddev Bandwidth: 9.31244
>> >> Max bandwidth (MB/sec): 148
>> > ^^^
>> >That wouldn't be possible from an external client.
>> >
>> >> Min bandwidth (MB/sec): 92
>> >> Average IOPS: 29
>> >> Stddev IOPS: 2
>> >> Max IOPS: 37
>> >> Min IOPS: 23
>> >> Average Latency(s): 0.544365
>> >> Stddev Latency(s): 0.35825
>> >> Max latency(s): 5.42548
>> >Very high max latency, telling us that your cluster ran out of steam at
>> >some point.
>> >
>> >> Min latency(s): 0.101533
>> >>
>> >> rados bench -p scbench 120 seq
>> >> Total time run: 120.880920
>> >> Total reads made: 1932
>> >> Read size: 4194304
>> >> Object size: 4194304
>> >> Bandwidth (MB/sec): 63.9307
>> >> Average IOPS 15
>> >> Stddev IOPS: 3
>> >> Max IOPS: 25
>> >> Min IOPS: 5
>> >> Average Latency(s): 0.999095
>> >> Max latency(s): 8.50774
>> >> Min latency(s): 0.0391591
>> >>
>> >> rados bench -p scbench 120 rand
>> >> Total time run: 121.059005
>> >> Total reads made: 1920
>> >> Read size: 4194304
>> >> Object size: 4194304
>> >> Bandwidth (MB/sec): 63.4401
>> >> Average IOPS: 15
>> >> Stddev IOPS: 4
>> >> Max IOPS: 26
>> >> Min IOPS: 1
>> >> Average Latency(s): 1.00785
>> >> Max latency(s): 6.48138
>> >> Min latency(s): 0.038925
>> >>
>> >> On XFS partitions fragmentation no more than 1%
>> >I'd de-frag anyway, just to rule that out.
>> >
>> >When doing your tests or normal (busy) operations from the client VM, run
>> >atop on your storage nodes and observe your OSD HDDs.
>> >Do they get busy, around 100%?
>> >
>> >Check with iperf or NPtcp that your network to the clients from the
>> >storage nodes is fully functional.
>> >
>> >Christian
>> >--
>> >Christian Balzer Network/Systems Engineer
>> > ch...@gol.com Global OnLine Japan/Rakuten Communications
>> > http://www.gol.com/
>>
>
>
>--
>Christian Balzer Network/Systems Engineer
>ch...@gol.com Global OnLine Japan/Rakuten Communications
>http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
