Hello Somnath, On Fri, Apr 24, 2015 at 04:23:19PM +0000, Somnath Roy wrote: > This could be again because of tcmalloc issue I reported earlier. > > Two things to observe. > > 1. Is the performance improving if you stop IO on other volume ? If so, it > could be different issue. there is no other IO.. only cephfs mounted, but no users of it.
> > 2. Run perf top in the OSD node and see if tcmalloc traces are popping up. don't see anything special: 3.34% libc-2.12.so [.] _int_malloc 2.87% libc-2.12.so [.] _int_free 2.79% [vdso] [.] __vdso_gettimeofday 2.67% libsoftokn3.so [.] 0x000000000001fad9 2.34% libfreeblpriv3.so [.] 0x00000000000355e6 2.33% libpthread-2.12.so [.] pthread_mutex_unlock 2.19% libpthread-2.12.so [.] pthread_mutex_lock 1.80% libc-2.12.so [.] malloc 1.43% [kernel] [k] do_raw_spin_lock 1.42% libc-2.12.so [.] memcpy 1.23% [kernel] [k] __switch_to 1.19% [kernel] [k] acpi_processor_ffh_cstate_enter 1.09% libc-2.12.so [.] malloc_consolidate 1.08% [kernel] [k] __schedule 1.05% libtcmalloc.so.4.1.0 [.] 0x0000000000017e6f 0.98% libc-2.12.so [.] vfprintf 0.83% libstdc++.so.6.0.13 [.] std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, 0.76% libstdc++.so.6.0.13 [.] 0x000000000008092a 0.73% libc-2.12.so [.] __memset_sse2 0.72% libc-2.12.so [.] __strlen_sse42 0.70% libstdc++.so.6.0.13 [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) 0.68% libpthread-2.12.so [.] pthread_mutex_trylock 0.67% librados.so.2.0.0 [.] ceph_crc32c_sctp 0.63% libpython2.6.so.1.0 [.] 0x000000000007d823 0.55% libnss3.so [.] 0x0000000000056d2a 0.52% libc-2.12.so [.] free 0.50% libstdc++.so.6.0.13 [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) should I check anything else? BR nik > > Thanks & Regards > Somnath > > -----Original Message----- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Nikola Ciprich > Sent: Friday, April 24, 2015 7:10 AM > To: ceph-users@lists.ceph.com > Cc: n...@linuxbox.cz > Subject: [ceph-users] very different performance on two volumes in the same > pool > > Hello, > > I'm trying to solve a bit mysterious situation: > > I've got 3 nodes CEPH cluster, and pool made of 3 OSDs (each on one node), > OSDs are 1TB SSD drives. > > pool has 3 replicas set. I'm measuring random IO performance using fio: > > fio --randrepeat=1 --ioengine=rbd --direct=1 --gtod_reduce=1 --name=test > --pool=ssd3r --rbdname=${rbdname} --invalidate=1 --bs=4k --iodepth=64 > --readwrite=randread --output=randio.log > > it's giving very nice performance of ~ 186K IOPS for random read. > > the problem is, I've got one volume on which it fives only ~20K IOPS and I > can't figure why. It's created using python, so I first suspected it can be > similar to missing layerign problem I was consulting here few days ago, but > when I tried reproducing it, I'm beting ~180K IOPS even for another volumes > created using python. > > so there is only this one problematic, others are fine. Since there is only > one SSD in each box and I'm using 3 replicas, there should not be any > difference in physical storage used between volumes.. > > I'm using hammer, 0.94.1, fio 2.2.6. > > here's RBD info: > > "slow" volume: > > [root@vfnphav1a fio]# rbd info ssd3r/vmtst23-6 rbd image 'vmtst23-6': > size 30720 MB in 7680 objects > order 22 (4096 kB objects) > block_name_prefix: rbd_data.1376d82ae8944a > format: 2 > features: > flags: > > "fast" volume: > [root@vfnphav1a fio]# rbd info ssd3r/vmtst23-7 rbd image 'vmtst23-7': > size 30720 MB in 7680 objects > order 22 (4096 kB objects) > block_name_prefix: rbd_data.13d01d2ae8944a > format: 2 > features: > flags: > > any idea on what could be wrong here? > > thanks a lot in advance! > > BR > > nik > > -- > ------------------------------------- > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28.rijna 168, 709 00 Ostrava > > tel.: +420 591 166 214 > fax: +420 596 621 273 > mobil: +420 777 093 799 > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: ser...@linuxbox.cz > ------------------------------------- > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly > prohibited. If you have received this communication in error, please notify > the sender by telephone or e-mail (as shown above) immediately and destroy > any and all copies of this message in your possession (whether hard copies or > electronically stored copies). > > -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz -------------------------------------
pgp9pONQaEDke.pgp
Description: PGP signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
