> Op 24 jun. 2017 om 14:17 heeft Maged Mokhtar <mmokh...@petasan.org> het > volgende geschreven: > > My understanding was this test is targeting latency more than IOPS. This is > probably why its was run using QD=1. It also makes sense that cpu freq will > be more important than cores. >
But then it is not generic enough to be used as an advise! It is just a line in 3D-space. As there are so many --WjW >> On 2017-06-24 12:52, Willem Jan Withagen wrote: >> >>> On 24-6-2017 05:30, Christian Wuerdig wrote: >>> The general advice floating around is that your want CPUs with high >>> clock speeds rather than more cores to reduce latency and increase IOPS >>> for SSD setups (see also >>> http://www.sys-pro.co.uk/ceph-storage-fast-cpus-ssd-performance/) So >>> something like a E5-2667V4 might bring better results in that situation. >>> Also there was some talk about disabling the processor C states in order >>> to bring latency down (something like this should be easy to test: >>> https://stackoverflow.com/a/22482722/220986) >> >> I would be very careful to call this a general advice... >> >> Although the article is interesting, it is rather single sided. >> >> The only thing is shows that there is a lineair relation between >> clockspeed and write or read speeds??? >> The article is rather vague on how and what is actually tested. >> >> By just running a single OSD with no replication a lot of the >> functionality is left out of the equation. >> Nobody is running just 1 osD on a box in a normal cluster host. >> >> Not using a serious SSD is another source of noise on the conclusion. >> More Queue depth can/will certainly have impact on concurrency. >> >> I would call this an observation, and nothing more. >> >> --WjW >>> >>> >>> On Sat, Jun 24, 2017 at 1:28 AM, Kostas Paraskevopoulos >>> <reverend...@gmail.com <mailto:reverend...@gmail.com>> wrote: >>> >>> Hello, >>> >>> We are in the process of evaluating the performance of a testing >>> cluster (3 nodes) with ceph jewel. Our setup consists of: >>> 3 monitors (VMs) >>> 2 physical servers each connected with 1 JBOD running Ubuntu Server >>> 16.04 >>> >>> Each server has 32 threads @2.1GHz and 128GB RAM. >>> The disk distribution per server is: >>> 38 * HUS726020ALS210 (SAS rotational) >>> 2 * HUSMH8010BSS200 (SAS SSD for journals) >>> 2 * ST1920FM0043 (SAS SSD for data) >>> 1 * INTEL SSDPEDME012T4 (NVME measured with fio ~300K iops) >>> >>> Since we don't currently have a 10Gbit switch, we test the performance >>> with the cluster in a degraded state, the noout flag set and we mount >>> rbd images on the powered on osd node. We confirmed that the network >>> is not saturated during the tests. >>> >>> We ran tests on the NVME disk and the pool created on this disk where >>> we hoped to get the most performance without getting limited by the >>> hardware specs since we have more disks than CPU threads. >>> >>> The nvme disk was at first partitioned with one partition and the >>> journal on the same disk. The performance on random 4K reads was >>> topped at 50K iops. We then removed the osd and partitioned with 4 >>> data partitions and 4 journals on the same disk. The performance >>> didn't increase significantly. Also, since we run read tests, the >>> journals shouldn't cause performance issues. >>> >>> We then ran 4 fio processes in parallel on the same rbd mounted image >>> and the total iops reached 100K. More parallel fio processes didn't >>> increase the measured iops. >>> >>> Our ceph.conf is pretty basic (debug is set to 0/0 for everything) and >>> the crushmap just defines the different buckets/rules for the disk >>> separation (rotational, ssd, nvme) in order to create the required >>> pools >>> >>> Is the performance of 100.000 iops for random 4K read normal for a >>> disk that on the same benchmark runs at more than 300K iops on the >>> same hardware or are we missing something? >>> >>> Best regards, >>> Kostas >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
