Hi ,
Looks like your cluster have warning message of "2 near full osd(s)".
Maybe try to extend osds first？
Best wishes,
Mika
2015-11-12 23:05 GMT+08:00 min fang <louisfang2...@gmail.com>:
> Hi cepher, I tried to use the following command to create a img, but
> unfortunately, the command hung for a long time until I broken it by
> crtl-z.
>
> rbd -p hello create img-003 --size 512
>
> so I checked the cluster status, and showed:
>
> cluster 0379cebd-b546-4954-b5d6-e13d08b7d2f1
> health HEALTH_WARN
> 2 near full osd(s)
> too many PGs per OSD (320 > max 300)
> monmap e2: 1 mons at {vl=192.168.90.253:6789/0}
> election epoch 1, quorum 0 vl
> osdmap e37: 2 osds: 2 up, 2 in
> pgmap v19544: 320 pgs, 3 pools, 12054 MB data, 3588 objects
> 657 GB used, 21867 MB / 714 GB avail
> 320 active+clean
>
> I did not see error message here could cause rbd create hang.
>
> I opened the client log and see:
>
> 2015-11-12 22:52:44.687491 7f89eced9780 20 librbd: create 0x7fff8f7b7800
> name = img-003 size = 536870912 old_format = 1 features = 0 order = 22
> stripe_unit = 0 stripe_count = 0
> 2015-11-12 22:52:44.687653 7f89eced9780 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6800/5472 -- osd_op(client.34321.0:1 img-003.rbd
> [stat] 2.8a047315 ack+read+known_if_redirected e37) v5 -- ?+0 0x28513d0 con
> 0x2850000
> 2015-11-12 22:52:44.688928 7f89e066b700 1 -- 192.168.90.253:0/1006121
> <== osd.1 192.168.90.253:6800/5472 1 ==== osd_op_reply(1 img-003.rbd
> [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 178+0+0
> (3550830125 0 0) 0x7f89c0000ae0 con 0x2850000
> 2015-11-12 22:52:44.689090 7f89eced9780 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- osd_op(client.34321.0:2 rbd_id.img-003
> [stat] 2.638c75a8 ack+read+known_if_redirected e37) v5 -- ?+0 0x2858330 con
> 0x2856f50
> 2015-11-12 22:52:44.690425 7f89e0469700 1 -- 192.168.90.253:0/1006121
> <== osd.0 192.168.90.253:6801/5344 1 ==== osd_op_reply(2 rbd_id.img-003
> [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 181+0+0
> (1202435393 0 0) 0x7f89b8000ae0 con 0x2856f50
> 2015-11-12 22:52:44.690494 7f89eced9780 2 librbd: adding rbd image to
> directory...
> 2015-11-12 22:52:44.690544 7f89eced9780 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- osd_op(client.34321.0:3 rbd_directory
> [tmapup 0~0] 2.30a98c1c ondisk+write+known_if_redirected e37) v5 -- ?+0
> 0x2858920 con 0x2856f50
> 2015-11-12 22:52:59.687447 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6789/0 -- mon_subscribe({monmap=3+,osdmap=38}) v2 --
> ?+0 0x7f89b0000ab0 con 0x2843b90
> 2015-11-12 22:52:59.687472 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000f40
> con 0x2856f50
> 2015-11-12 22:52:59.687887 7f89e3873700 1 -- 192.168.90.253:0/1006121
> <== mon.0 192.168.90.253:6789/0 11 ==== mon_subscribe_ack(300s) v1 ====
> 20+0+0 (2867606018 0 0) 0x7f89d8001160 con 0x2843b90
> 2015-11-12 22:53:04.687593 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:09.687731 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:14.687844 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:19.687978 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:24.688116 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:29.688253 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:34.688389 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:39.688512 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
> 2015-11-12 22:53:44.688636 7f89e4074700 1 -- 192.168.90.253:0/1006121
> --> 192.168.90.253:6801/5344 -- ping magic: 0 v1 -- ?+0 0x7f89b0000ab0
> con 0x2856f50
>
>
> Looks to me, we are keeping ping magic process, and no completed.
>
> my ceph version is "ceph version 0.94.5
> (9764da52395923e0b32908d83a9f7304401fee43)"
>
> somebody can help me on this? Or still me to collect more debug
> information for analyzing.
>
> Thanks.
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
