>>We should discuss this at the perf meeting today. We knew NUMA node >>affinity issues were going to crop up sooner or later (and indeed >>already have in some cases), but this is pretty major. It's probably >>time to really dig in and figure out how to deal with this.
Damn, I'm on the road currently,I'll be to short for me to going to the meting today. Sorry. >>Note: this is one of the reasons I like small nodes with single sockets >>and fewer OSDs. Well, Indeed it's a 2 sockets nodes, but with only 4 osd currently. ----- Mail original ----- De: "Mark Nelson" <mnel...@redhat.com> À: "aderumier" <aderum...@odiso.com>, "Milosz Tanski" <mil...@adfin.com> Cc: "ceph-devel" <ceph-de...@vger.kernel.org>, "ceph-users" <ceph-users@lists.ceph.com> Envoyé: Mercredi 22 Avril 2015 16:01:12 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi Alexandre, We should discuss this at the perf meeting today. We knew NUMA node affinity issues were going to crop up sooner or later (and indeed already have in some cases), but this is pretty major. It's probably time to really dig in and figure out how to deal with this. Note: this is one of the reasons I like small nodes with single sockets and fewer OSDs. Mark On 04/22/2015 08:56 AM, Alexandre DERUMIER wrote: > Hi, > > I have done a lot of test today, and it seem indeed numa related. > > My numastat was > > # numastat > node0 node1 > numa_hit 99075422 153976877 > numa_miss 167490965 1493663 > numa_foreign 1493663 167491417 > interleave_hit 157745 167015 > local_node 99049179 153830554 > other_node 167517697 1639986 > > So, a lot of miss. > > In this case , I can reproduce ios going from 85k to 300k iops, up and down. > > now setting > echo 0 > /proc/sys/kernel/numa_balancing > > and starting osd daemons with > > numactl --interleave=all /usr/bin/ceph-osd > > > I have a constant 300k iops ! > > > I wonder if it could be improve by binding osd daemons to specific numa node. > I have 2 numanode of 10 cores with 6 osd, but I think it also require > ceph.conf osd threads tunning. > > > > ----- Mail original ----- > De: "Milosz Tanski" <mil...@adfin.com> > À: "aderumier" <aderum...@odiso.com> > Cc: "ceph-devel" <ceph-de...@vger.kernel.org>, "ceph-users" > <ceph-users@lists.ceph.com> > Envoyé: Mercredi 22 Avril 2015 12:54:23 > Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon > improve performance from 100k iops to 300k iops > > > > On Wed, Apr 22, 2015 at 5:01 AM, Alexandre DERUMIER < aderum...@odiso.com > > wrote: > > > I wonder if it could be numa related, > > I'm using centos 7.1, > and auto numa balacning is enabled > > cat /proc/sys/kernel/numa_balancing = 1 > > Maybe osd daemon access to buffer on wrong numa node. > > I'll try to reproduce the problem > > > > Can you force the degenerate case using numactl? To either affirm or deny > your suspicion. > > > > > ----- Mail original ----- > De: "aderumier" < aderum...@odiso.com > > À: "ceph-devel" < ceph-de...@vger.kernel.org >, "ceph-users" < > ceph-users@lists.ceph.com > > Envoyé: Mercredi 22 Avril 2015 10:40:05 > Objet: [ceph-users] strange benchmark problem : restarting osd daemon improve > performance from 100k iops to 300k iops > > Hi, > > I was doing some benchmarks, > I have found an strange behaviour. > > Using fio with rbd engine, I was able to reach around 100k iops. > (osd datas in linux buffer, iostat show 0% disk access) > > then after restarting all osd daemons, > > the same fio benchmark show now around 300k iops. > (osd datas in linux buffer, iostat show 0% disk access) > > > any ideas? > > > > > before restarting osd > --------------------- > rbd_iodepth32-test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, > iodepth=32 > ... > fio-2.2.7-10-g51e9 > Starting 10 processes > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > ^Cbs: 10 (f=10): [r(10)] [2.9% done] [376.1MB/0KB/0KB /s] [96.6K/0/0 iops] > [eta 14m:45s] > fio: terminating on signal 2 > > rbd_iodepth32-test: (groupid=0, jobs=10): err= 0: pid=17075: Wed Apr 22 > 10:00:04 2015 > read : io=11558MB, bw=451487KB/s, iops=112871, runt= 26215msec > slat (usec): min=5, max=3685, avg=16.89, stdev=17.38 > clat (usec): min=5, max=62584, avg=2695.80, stdev=5351.23 > lat (usec): min=109, max=62598, avg=2712.68, stdev=5350.42 > clat percentiles (usec): > | 1.00th=[ 155], 5.00th=[ 183], 10.00th=[ 205], 20.00th=[ 247], > | 30.00th=[ 294], 40.00th=[ 354], 50.00th=[ 446], 60.00th=[ 660], > | 70.00th=[ 1176], 80.00th=[ 3152], 90.00th=[ 9024], 95.00th=[14656], > | 99.00th=[25984], 99.50th=[30336], 99.90th=[38656], 99.95th=[41728], > | 99.99th=[47360] > bw (KB /s): min=23928, max=154416, per=10.07%, avg=45462.82, stdev=28809.95 > lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=20.79% > lat (usec) : 500=32.74%, 750=8.99%, 1000=5.03% > lat (msec) : 2=8.37%, 4=6.21%, 10=8.90%, 20=6.60%, 50=2.37% > lat (msec) : 100=0.01% > cpu : usr=15.90%, sys=3.01%, ctx=765446, majf=0, minf=8710 > IO depths : 1=0.4%, 2=0.9%, 4=2.3%, 8=7.4%, 16=75.5%, 32=13.6%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=93.6%, 8=2.8%, 16=2.4%, 32=1.2%, 64=0.0%, >=64=0.0% > issued : total=r=2958935/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=11558MB, aggrb=451487KB/s, minb=451487KB/s, maxb=451487KB/s, > mint=26215msec, maxt=26215msec > > Disk stats (read/write): > sdg: ios=0/29, merge=0/16, ticks=0/3, in_queue=3, util=0.01% > [root@ceph1-3 fiorbd]# ./fio fiorbd > rbd_iodepth32-test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, > iodepth=32 > > > > > AFTER RESTARTING OSDS > ---------------------- > [root@ceph1-3 fiorbd]# ./fio fiorbd > rbd_iodepth32-test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, > iodepth=32 > ... > fio-2.2.7-10-g51e9 > Starting 10 processes > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > rbd engine: RBD version: 0.1.9 > ^Cbs: 10 (f=10): [r(10)] [0.2% done] [1155MB/0KB/0KB /s] [296K/0/0 iops] [eta > 01h:09m:27s] > fio: terminating on signal 2 > > rbd_iodepth32-test: (groupid=0, jobs=10): err= 0: pid=18252: Wed Apr 22 > 10:02:28 2015 > read : io=7655.7MB, bw=1036.8MB/s, iops=265218, runt= 7389msec > slat (usec): min=5, max=3406, avg=26.59, stdev=40.35 > clat (usec): min=8, max=684328, avg=930.43, stdev=6419.12 > lat (usec): min=154, max=684342, avg=957.02, stdev=6419.28 > clat percentiles (usec): > | 1.00th=[ 243], 5.00th=[ 314], 10.00th=[ 366], 20.00th=[ 450], > | 30.00th=[ 524], 40.00th=[ 604], 50.00th=[ 692], 60.00th=[ 796], > | 70.00th=[ 924], 80.00th=[ 1096], 90.00th=[ 1400], 95.00th=[ 1720], > | 99.00th=[ 2672], 99.50th=[ 3248], 99.90th=[ 5920], 99.95th=[ 9792], > | 99.99th=[436224] > bw (KB /s): min=32614, max=143160, per=10.19%, avg=108076.46, stdev=28263.82 > lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=1.23% > lat (usec) : 500=25.64%, 750=29.15%, 1000=18.84% > lat (msec) : 2=22.19%, 4=2.69%, 10=0.21%, 20=0.02%, 50=0.01% > lat (msec) : 250=0.01%, 500=0.02%, 750=0.01% > cpu : usr=44.06%, sys=11.26%, ctx=642620, majf=0, minf=6832 > IO depths : 1=0.1%, 2=0.5%, 4=2.0%, 8=11.5%, 16=77.8%, 32=8.1%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% > complete : 0=0.0%, 4=94.1%, 8=1.3%, 16=2.3%, 32=2.3%, 64=0.0%, >=64=0.0% > issued : total=r=1959697/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 > latency : target=0, window=0, percentile=100.00%, depth=32 > > Run status group 0 (all jobs): > READ: io=7655.7MB, aggrb=1036.8MB/s, minb=1036.8MB/s, maxb=1036.8MB/s, > mint=7389msec, maxt=7389msec > > Disk stats (read/write): > sdg: ios=0/21, merge=0/10, ticks=0/2, in_queue=2, util=0.03% > > > > > CEPH LOG > -------- > > before restarting osd > ---------------------- > > 2015-04-22 09:53:17.568095 mon.0 10.7.0.152:6789/0 2144 : cluster [INF] pgmap > v11330: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 298 MB/s rd, > 76465 op/s > 2015-04-22 09:53:18.574524 mon.0 10.7.0.152:6789/0 2145 : cluster [INF] pgmap > v11331: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 333 MB/s rd, > 85355 op/s > 2015-04-22 09:53:19.579351 mon.0 10.7.0.152:6789/0 2146 : cluster [INF] pgmap > v11332: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 343 MB/s rd, > 87932 op/s > 2015-04-22 09:53:20.591586 mon.0 10.7.0.152:6789/0 2147 : cluster [INF] pgmap > v11333: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 328 MB/s rd, > 84151 op/s > 2015-04-22 09:53:21.600650 mon.0 10.7.0.152:6789/0 2148 : cluster [INF] pgmap > v11334: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 237 MB/s rd, > 60855 op/s > 2015-04-22 09:53:22.607966 mon.0 10.7.0.152:6789/0 2149 : cluster [INF] pgmap > v11335: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 144 MB/s rd, > 36935 op/s > 2015-04-22 09:53:23.617780 mon.0 10.7.0.152:6789/0 2150 : cluster [INF] pgmap > v11336: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 321 MB/s rd, > 82334 op/s > 2015-04-22 09:53:24.622341 mon.0 10.7.0.152:6789/0 2151 : cluster [INF] pgmap > v11337: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 368 MB/s rd, > 94211 op/s > 2015-04-22 09:53:25.628432 mon.0 10.7.0.152:6789/0 2152 : cluster [INF] pgmap > v11338: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 244 MB/s rd, > 62644 op/s > 2015-04-22 09:53:26.632855 mon.0 10.7.0.152:6789/0 2153 : cluster [INF] pgmap > v11339: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 175 MB/s rd, > 44997 op/s > 2015-04-22 09:53:27.636573 mon.0 10.7.0.152:6789/0 2154 : cluster [INF] pgmap > v11340: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 122 MB/s rd, > 31259 op/s > 2015-04-22 09:53:28.645784 mon.0 10.7.0.152:6789/0 2155 : cluster [INF] pgmap > v11341: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 229 MB/s rd, > 58674 op/s > 2015-04-22 09:53:29.657128 mon.0 10.7.0.152:6789/0 2156 : cluster [INF] pgmap > v11342: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 271 MB/s rd, > 69501 op/s > 2015-04-22 09:53:30.662796 mon.0 10.7.0.152:6789/0 2157 : cluster [INF] pgmap > v11343: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 211 MB/s rd, > 54020 op/s > 2015-04-22 09:53:31.666421 mon.0 10.7.0.152:6789/0 2158 : cluster [INF] pgmap > v11344: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 164 MB/s rd, > 42001 op/s > 2015-04-22 09:53:32.670842 mon.0 10.7.0.152:6789/0 2159 : cluster [INF] pgmap > v11345: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 134 MB/s rd, > 34380 op/s > 2015-04-22 09:53:33.681357 mon.0 10.7.0.152:6789/0 2160 : cluster [INF] pgmap > v11346: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 293 MB/s rd, > 75213 op/s > 2015-04-22 09:53:34.692177 mon.0 10.7.0.152:6789/0 2161 : cluster [INF] pgmap > v11347: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 337 MB/s rd, > 86353 op/s > 2015-04-22 09:53:35.697401 mon.0 10.7.0.152:6789/0 2162 : cluster [INF] pgmap > v11348: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 229 MB/s rd, > 58839 op/s > 2015-04-22 09:53:36.699309 mon.0 10.7.0.152:6789/0 2163 : cluster [INF] pgmap > v11349: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 390 GB data, 391 GB used, 904 GB / 1295 GB avail; 152 MB/s rd, > 39117 op/s > > > restarting osd > --------------- > > 2015-04-22 10:00:09.766906 mon.0 10.7.0.152:6789/0 2255 : cluster [INF] osd.0 > marked itself down > 2015-04-22 10:00:09.790212 mon.0 10.7.0.152:6789/0 2256 : cluster [INF] > osdmap e849: 9 osds: 8 up, 9 in > 2015-04-22 10:00:09.793050 mon.0 10.7.0.152:6789/0 2257 : cluster [INF] pgmap > v11439: 964 pgs: 2 active+undersized+degraded, 8 stale+active+remapped, 106 > stale+active+clean, 54 active+remapped, 794 active+clean; 419 GB data, 420 GB > used, 874 GB / 1295 GB avail; 516 kB/s rd, 130 op/s > 2015-04-22 10:00:10.795966 mon.0 10.7.0.152:6789/0 2258 : cluster [INF] > osdmap e850: 9 osds: 8 up, 9 in > 2015-04-22 10:00:10.796675 mon.0 10.7.0.152:6789/0 2259 : cluster [INF] pgmap > v11440: 964 pgs: 2 active+undersized+degraded, 8 stale+active+remapped, 106 > stale+active+clean, 54 active+remapped, 794 active+clean; 419 GB data, 420 GB > used, 874 GB / 1295 GB avail > 2015-04-22 10:00:11.798257 mon.0 10.7.0.152:6789/0 2260 : cluster [INF] pgmap > v11441: 964 pgs: 2 active+undersized+degraded, 8 stale+active+remapped, 106 > stale+active+clean, 54 active+remapped, 794 active+clean; 419 GB data, 420 GB > used, 874 GB / 1295 GB avail > 2015-04-22 10:00:12.339696 mon.0 10.7.0.152:6789/0 2262 : cluster [INF] osd.1 > marked itself down > 2015-04-22 10:00:12.800168 mon.0 10.7.0.152:6789/0 2263 : cluster [INF] > osdmap e851: 9 osds: 7 up, 9 in > 2015-04-22 10:00:12.806498 mon.0 10.7.0.152:6789/0 2264 : cluster [INF] pgmap > v11443: 964 pgs: 1 active+undersized+degraded, 13 stale+active+remapped, 216 > stale+active+clean, 49 active+remapped, 684 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:13.804186 mon.0 10.7.0.152:6789/0 2265 : cluster [INF] > osdmap e852: 9 osds: 7 up, 9 in > 2015-04-22 10:00:13.805216 mon.0 10.7.0.152:6789/0 2266 : cluster [INF] pgmap > v11444: 964 pgs: 1 active+undersized+degraded, 13 stale+active+remapped, 216 > stale+active+clean, 49 active+remapped, 684 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:14.781785 mon.0 10.7.0.152:6789/0 2268 : cluster [INF] osd.2 > marked itself down > 2015-04-22 10:00:14.810571 mon.0 10.7.0.152:6789/0 2269 : cluster [INF] > osdmap e853: 9 osds: 6 up, 9 in > 2015-04-22 10:00:14.813871 mon.0 10.7.0.152:6789/0 2270 : cluster [INF] pgmap > v11445: 964 pgs: 1 active+undersized+degraded, 22 stale+active+remapped, 300 > stale+active+clean, 40 active+remapped, 600 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:15.810333 mon.0 10.7.0.152:6789/0 2271 : cluster [INF] > osdmap e854: 9 osds: 6 up, 9 in > 2015-04-22 10:00:15.811425 mon.0 10.7.0.152:6789/0 2272 : cluster [INF] pgmap > v11446: 964 pgs: 1 active+undersized+degraded, 22 stale+active+remapped, 300 > stale+active+clean, 40 active+remapped, 600 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:16.395105 mon.0 10.7.0.152:6789/0 2273 : cluster [INF] > HEALTH_WARN; 2 pgs degraded; 323 pgs stale; 2 pgs stuck degraded; 64 pgs > stuck unclean; 2 pgs stuck undersized; 2 pgs undersized; 3/9 in osds are > down; clock skew detected on mon.ceph1-2 > 2015-04-22 10:00:16.814432 mon.0 10.7.0.152:6789/0 2274 : cluster [INF] osd.1 > 10.7.0.152:6800/14848 boot > 2015-04-22 10:00:16.814938 mon.0 10.7.0.152:6789/0 2275 : cluster [INF] > osdmap e855: 9 osds: 7 up, 9 in > 2015-04-22 10:00:16.815942 mon.0 10.7.0.152:6789/0 2276 : cluster [INF] pgmap > v11447: 964 pgs: 1 active+undersized+degraded, 22 stale+active+remapped, 300 > stale+active+clean, 40 active+remapped, 600 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:17.222281 mon.0 10.7.0.152:6789/0 2278 : cluster [INF] osd.3 > marked itself down > 2015-04-22 10:00:17.819371 mon.0 10.7.0.152:6789/0 2279 : cluster [INF] > osdmap e856: 9 osds: 6 up, 9 in > 2015-04-22 10:00:17.822041 mon.0 10.7.0.152:6789/0 2280 : cluster [INF] pgmap > v11448: 964 pgs: 1 active+undersized+degraded, 25 stale+active+remapped, 394 > stale+active+clean, 37 active+remapped, 506 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:18.551068 mon.0 10.7.0.152:6789/0 2282 : cluster [INF] osd.6 > marked itself down > 2015-04-22 10:00:18.819387 mon.0 10.7.0.152:6789/0 2283 : cluster [INF] osd.2 > 10.7.0.152:6812/15410 boot > 2015-04-22 10:00:18.821134 mon.0 10.7.0.152:6789/0 2284 : cluster [INF] > osdmap e857: 9 osds: 6 up, 9 in > 2015-04-22 10:00:18.824440 mon.0 10.7.0.152:6789/0 2285 : cluster [INF] pgmap > v11449: 964 pgs: 1 active+undersized+degraded, 30 stale+active+remapped, 502 > stale+active+clean, 32 active+remapped, 398 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:19.820947 mon.0 10.7.0.152:6789/0 2287 : cluster [INF] > osdmap e858: 9 osds: 6 up, 9 in > 2015-04-22 10:00:19.821853 mon.0 10.7.0.152:6789/0 2288 : cluster [INF] pgmap > v11450: 964 pgs: 1 active+undersized+degraded, 30 stale+active+remapped, 502 > stale+active+clean, 32 active+remapped, 398 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:20.828047 mon.0 10.7.0.152:6789/0 2290 : cluster [INF] osd.3 > 10.7.0.152:6816/15971 boot > 2015-04-22 10:00:20.828431 mon.0 10.7.0.152:6789/0 2291 : cluster [INF] > osdmap e859: 9 osds: 7 up, 9 in > 2015-04-22 10:00:20.829126 mon.0 10.7.0.152:6789/0 2292 : cluster [INF] pgmap > v11451: 964 pgs: 1 active+undersized+degraded, 30 stale+active+remapped, 502 > stale+active+clean, 32 active+remapped, 398 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:20.991343 mon.0 10.7.0.152:6789/0 2294 : cluster [INF] osd.7 > marked itself down > 2015-04-22 10:00:21.830389 mon.0 10.7.0.152:6789/0 2295 : cluster [INF] osd.0 > 10.7.0.152:6804/14481 boot > 2015-04-22 10:00:21.832518 mon.0 10.7.0.152:6789/0 2296 : cluster [INF] > osdmap e860: 9 osds: 7 up, 9 in > 2015-04-22 10:00:21.836129 mon.0 10.7.0.152:6789/0 2297 : cluster [INF] pgmap > v11452: 964 pgs: 1 active+undersized+degraded, 35 stale+active+remapped, 608 > stale+active+clean, 27 active+remapped, 292 active+clean, 1 > stale+active+undersized+degraded; 419 GB data, 420 GB used, 874 GB / 1295 GB > avail > 2015-04-22 10:00:22.830456 mon.0 10.7.0.152:6789/0 2298 : cluster [INF] osd.6 > 10.7.0.153:6808/21955 boot > 2015-04-22 10:00:22.832171 mon.0 10.7.0.152:6789/0 2299 : cluster [INF] > osdmap e861: 9 osds: 8 up, 9 in > 2015-04-22 10:00:22.836272 mon.0 10.7.0.152:6789/0 2300 : cluster [INF] pgmap > v11453: 964 pgs: 3 active+undersized+degraded, 27 stale+active+remapped, 498 > stale+active+clean, 2 peering, 28 active+remapped, 402 active+clean, 4 > remapped+peering; 419 GB data, 420 GB used, 874 GB / 1295 GB avail > 2015-04-22 10:00:23.420309 mon.0 10.7.0.152:6789/0 2302 : cluster [INF] osd.8 > marked itself down > 2015-04-22 10:00:23.833708 mon.0 10.7.0.152:6789/0 2303 : cluster [INF] > osdmap e862: 9 osds: 7 up, 9 in > 2015-04-22 10:00:23.836459 mon.0 10.7.0.152:6789/0 2304 : cluster [INF] pgmap > v11454: 964 pgs: 3 active+undersized+degraded, 44 stale+active+remapped, 587 > stale+active+clean, 2 peering, 11 active+remapped, 313 active+clean, 4 > remapped+peering; 419 GB data, 420 GB used, 874 GB / 1295 GB avail > 2015-04-22 10:00:24.832905 mon.0 10.7.0.152:6789/0 2305 : cluster [INF] osd.7 > 10.7.0.153:6804/22536 boot > 2015-04-22 10:00:24.834381 mon.0 10.7.0.152:6789/0 2306 : cluster [INF] > osdmap e863: 9 osds: 8 up, 9 in > 2015-04-22 10:00:24.836977 mon.0 10.7.0.152:6789/0 2307 : cluster [INF] pgmap > v11455: 964 pgs: 3 active+undersized+degraded, 31 stale+active+remapped, 503 > stale+active+clean, 4 active+undersized+degraded+remapped, 5 peering, 13 > active+remapped, 397 active+clean, 8 remapped+peering; 419 GB data, 420 GB > used, 874 GB / 1295 GB avail > 2015-04-22 10:00:25.834459 mon.0 10.7.0.152:6789/0 2309 : cluster [INF] > osdmap e864: 9 osds: 8 up, 9 in > 2015-04-22 10:00:25.835727 mon.0 10.7.0.152:6789/0 2310 : cluster [INF] pgmap > v11456: 964 pgs: 3 active+undersized+degraded, 31 stale+active+remapped, 503 > stale+active+clean, 4 active+undersized+degraded+remapped, 5 peering, 13 > active > > > AFTER OSD RESTART > ------------------ > > > 2015-04-22 10:09:27.609052 mon.0 10.7.0.152:6789/0 2339 : cluster [INF] pgmap > v11478: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 786 MB/s rd, > 196 kop/s > 2015-04-22 10:09:28.618082 mon.0 10.7.0.152:6789/0 2340 : cluster [INF] pgmap > v11479: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 1578 MB/s rd, > 394 kop/s > 2015-04-22 10:09:30.629067 mon.0 10.7.0.152:6789/0 2341 : cluster [INF] pgmap > v11480: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 932 MB/s rd, > 233 kop/s > 2015-04-22 10:09:32.645890 mon.0 10.7.0.152:6789/0 2342 : cluster [INF] pgmap > v11481: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 627 MB/s rd, > 156 kop/s > 2015-04-22 10:09:33.652634 mon.0 10.7.0.152:6789/0 2343 : cluster [INF] pgmap > v11482: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 1034 MB/s rd, > 258 kop/s > 2015-04-22 10:09:35.655657 mon.0 10.7.0.152:6789/0 2344 : cluster [INF] pgmap > v11483: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 529 MB/s rd, > 132 kop/s > 2015-04-22 10:09:37.674332 mon.0 10.7.0.152:6789/0 2345 : cluster [INF] pgmap > v11484: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 770 MB/s rd, > 192 kop/s > 2015-04-22 10:09:38.679445 mon.0 10.7.0.152:6789/0 2346 : cluster [INF] pgmap > v11485: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 1358 MB/s rd, > 339 kop/s > 2015-04-22 10:09:40.690037 mon.0 10.7.0.152:6789/0 2347 : cluster [INF] pgmap > v11486: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 649 MB/s rd, > 162 kop/s > 2015-04-22 10:09:42.707164 mon.0 10.7.0.152:6789/0 2348 : cluster [INF] pgmap > v11487: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 580 MB/s rd, > 145 kop/s > 2015-04-22 10:09:43.713736 mon.0 10.7.0.152:6789/0 2349 : cluster [INF] pgmap > v11488: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 962 MB/s rd, > 240 kop/s > 2015-04-22 10:09:45.718658 mon.0 10.7.0.152:6789/0 2350 : cluster [INF] pgmap > v11489: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 506 MB/s rd, > 126 kop/s > 2015-04-22 10:09:47.737358 mon.0 10.7.0.152:6789/0 2351 : cluster [INF] pgmap > v11490: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 774 MB/s rd, > 193 kop/s > 2015-04-22 10:09:48.743338 mon.0 10.7.0.152:6789/0 2352 : cluster [INF] pgmap > v11491: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 1363 MB/s rd, > 340 kop/s > 2015-04-22 10:09:50.746685 mon.0 10.7.0.152:6789/0 2353 : cluster [INF] pgmap > v11492: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 662 MB/s rd, > 165 kop/s > 2015-04-22 10:09:52.762461 mon.0 10.7.0.152:6789/0 2354 : cluster [INF] pgmap > v11493: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 593 MB/s rd, > 148 kop/s > 2015-04-22 10:09:53.767729 mon.0 10.7.0.152:6789/0 2355 : cluster [INF] pgmap > v11494: 964 pgs: 2 active+undersized+degraded, 62 active+remapped, 900 > active+clean; 419 GB data, 421 GB used, 874 GB / 1295 GB avail; 938 MB/s rd, > 234 kop/s > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
