We are using librbd on a host with CentOS 7.2 via virtio-blk. This server hosts the VMs on which we are doing our tests. But we have exactly the same behaviour than #9071. We try to follow the thread to the bug 8818 but we didn't reproduce the issue with a lot of DD. Each time we try with mkfs.ext4, there is always one process over the 16 (we have 16 volumes) which hangs !
2017-01-16 17:45 GMT+01:00 Jason Dillaman <jdill...@redhat.com>: > Are you using krbd directly within the VM or librbd via > virtio-blk/scsi? Ticket #9071 is against krbd. > > On Mon, Jan 16, 2017 at 11:34 AM, Vincent Godin <vince.ml...@gmail.com> > wrote: > > In fact, we can reproduce the problem from VM with CentOS 6.7, 7.2 or > 7.3. > > We can reproduce it each time with this config : one VM (here in CentOS > 6.7) > > with 16 RBD volumes of 100GB attached. When we launch in serial > mkfs.ext4 on > > each of these volumes, we allways encounter the problem on one of them. > We > > tried with the option -E nodiscard but we still have the problem. It' > look > > exactly like the bug #9071 with the same dmesg message : > > > > vdh: unknown partition table > > EXT4-fs (vdf): mounted filesystem with ordered data mode. Opts: > > EXT4-fs (vdg): mounted filesystem with ordered data mode. Opts: > > INFO: task flush-252:112:2903 blocked for more than 120 seconds. > > Not tainted 2.6.32-573.18.1.el6.x86_64 #1 > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this > message. > > flush-252:112 D 0000000000000000 0 2903 2 0x00000080 > > ffff8808328bf6e0 0000000000000046 ffff8808ffffffff 000000003d697f73 > > 0000000000000000 ffff88082fbd7ec0 0000000000021454 ffffffffa78356ec > > 000000002b9db4fe ffffffff81aa6700 ffff88082efc9ad8 ffff8808328bffd8 > > Call Trace: > > [<ffffffff81539673>] io_schedule+0x73/0xc0 > > [<ffffffff81276598>] get_request_wait+0x108/0x1d0 > > [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 > > [<ffffffff812766f9>] blk_queue_bio+0x99/0x610 > > [<ffffffff81274ec0>] generic_make_request+0x240/0x5a0 > > [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 > > [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 > > [<ffffffff81275290>] submit_bio+0x70/0x120 > > [<ffffffff811c7dcd>] submit_bh+0x11d/0x1f0 > > [<ffffffff811ca588>] __block_write_full_page+0x1c8/0x330 > > [<ffffffff811c9550>] ? end_buffer_async_write+0x0/0x190 > > [<ffffffff811ce450>] ? blkdev_get_block+0x0/0x20 > > [<ffffffff811ce450>] ? blkdev_get_block+0x0/0x20 > > [<ffffffff811ca7d0>] block_write_full_page_endio+0xe0/0x120 > > [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130 > > [<ffffffff811ca825>] block_write_full_page+0x15/0x20 > > [<ffffffff811cf5e8>] blkdev_writepage+0x18/0x20 > > [<ffffffff8113b387>] __writepage+0x17/0x40 > > [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0 > > [<ffffffff8113b370>] ? __writepage+0x0/0x40 > > [<ffffffff8113c934>] generic_writepages+0x24/0x30 > > [<ffffffff8113c961>] do_writepages+0x21/0x40 > > [<ffffffff811bf01d>] writeback_single_inode+0xdd/0x290 > > [<ffffffff811bf41d>] writeback_sb_inodes+0xbd/0x170 > > [<ffffffff811bf57b>] writeback_inodes_wb+0xab/0x1b0 > > [<ffffffff811bf973>] wb_writeback+0x2f3/0x410 > > [<ffffffff811bfb4b>] wb_do_writeback+0xbb/0x240 > > [<ffffffff811bfd33>] bdi_writeback_task+0x63/0x1b0 > > [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0 > > [<ffffffff8114b760>] ? bdi_start_fn+0x0/0x100 > > [<ffffffff8114b7e6>] bdi_start_fn+0x86/0x100 > > [<ffffffff8114b760>] ? bdi_start_fn+0x0/0x100 > > [<ffffffff810a0fce>] kthread+0x9e/0xc0 > > [<ffffffff8100c28a>] child_rip+0xa/0x20 > > [<ffffffff810a0f30>] ? kthread+0x0/0xc0 > > [<ffffffff8100c280>] ? child_rip+0x0/0x20 > > INFO: task mkfs.ext4:3040 blocked for more than 120 seconds. > > Not tainted 2.6.32-573.18.1.el6.x86_64 #1 > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this > message. > > mkfs.ext4 D 0000000000000002 0 3040 3038 0x00000080 > > ffff88075e79f4d8 0000000000000082 ffff8808ffffffff 000000003d697f73 > > 0000000000000000 ffff88082fb73130 0000000000021472 ffffffffa78356ec > > 000000002b9db4fe ffffffff81aa6700 ffff88082e787068 ffff88075e79ffd8 > > Call Trace: > > [<ffffffff81539673>] io_schedule+0x73/0xc0 > > [<ffffffff81276598>] get_request_wait+0x108/0x1d0 > > [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 > > [<ffffffff812766f9>] blk_queue_bio+0x99/0x610 > > > > Ceph version is Jewel 10.2.3 > > Ceph clients, mons and servers have the kernel > 3.10.0-327.36.3.el7.x86_64 > > on CentOS 7.2 > > > > 2017-01-13 20:07 GMT+01:00 Jason Dillaman <jdill...@redhat.com>: > >> > >> You might be hitting this issue [1] where mkfs is issuing lots of > >> discard operations. If you get a chance, can you retest w/ the "-E > >> nodiscard" option? > >> > >> Thanks > >> > >> [1] http://tracker.ceph.com/issues/16689 > >> > >> On Fri, Jan 13, 2017 at 12:57 PM, Vincent Godin <vince.ml...@gmail.com> > >> wrote: > >> > Thanks Jason, > >> > > >> > We observed a curious behavior : we have some VMs on CentOS 6.x hosted > >> > on > >> > our Openstack computes which are in CentOS 7.2. If we try to make a > >> > mkfs.ext4 on a volume create with the Jewel default (61) on the VM > it's > >> > hung > >> > and we have to reboot the VM to get a responsive system. This is > strange > >> > because the libvirt process is launched from the host which is in > CentOS > >> > 7.2. If a disable some features, the mkfs.ext4 succeed. If the VM is > in > >> > CentOS 7.x, there is no probleme at all. Maybe the kernel of the > CentOS > >> > 6.X > >> > is unable to use the exclusive-lock feature ? > >> > I think we will have to stay in a very conservative > rbd_default_features > >> > such 1 because we don't use stripping and the others features are not > >> > compatible with our old CentOS 6.x VMs .. > >> > > >> > A last question : is the rbd object-map rebuild a long process ? in an > >> > other > >> > way, does it cost the same time as a delete (which read all the blocks > >> > possible for an image without omap feature). Is it a good idea to > enable > >> > omap feature on an already used image ? (I know that during the > rebuild > >> > process, the VM will have to be stopped) > >> > > >> > > >> > > >> > 2017-01-13 15:09 GMT+01:00 Jason Dillaman <jdill...@redhat.com>: > >> >> > >> >> On Fri, Jan 13, 2017 at 5:11 AM, Vincent Godin < > vince.ml...@gmail.com> > >> >> wrote: > >> >> > We are using a production cluster which started in Firefly, then > >> >> > moved > >> >> > to > >> >> > Giant, Hammer and finally Jewel. So our images have different > >> >> > features > >> >> > correspondind to the value of "rbd_default_features" of the version > >> >> > when > >> >> > they were created. > >> >> > We have actually three pack of features activated : > >> >> > image with : > >> >> > - layering ~ 1 > >> >> > - layering, striping ~3 > >> >> > - layering, exclusive-lock, object-map, fast-diff, deep-flatten ~ > 61 > >> >> > > >> >> > 1) Is it a good idea to try to give all images the same features ? > >> >> > >> >> It isn't needed. > >> >> > >> >> > 2) Is it possible to disable the striping feature on an already > >> >> > created > >> >> > image (we never specify any stripe-unit nor stripe-count) ? > >> >> > >> >> Negative -- striping cannot be dynamically disabled because it would > >> >> result in potentially altering the structure and placement of the > data > >> >> within the image. If your stripe-unit is the object size and the > >> >> stripe count is 1, that's a special case where the flag is > essentially > >> >> ignored. > >> >> > >> >> > 3) What is the behaviour of an already created image on which we > >> >> > activate > >> >> > the object-map feature ? Will a process try to rebuild a index of > >> >> > used > >> >> > blocks - if no, if we delete later the image, will ceph try to > remove > >> >> > all > >> >> > the blocks or only the blocks refered by object-map index ? > >> >> > >> >> You would need to run "rbd object-map rebuild <image-spec>" to > rebuild > >> >> the object map. Until it is rebuilt, it will be considered invalid > and > >> >> won't be used for reference. You can determine the object map state > by > >> >> running "rbd info <image-spec>" > >> >> > >> >> > 4) We are on Jewel but with tunables set to hammer (Centos 7.2). > What > >> >> > are > >> >> > the best default features to set in that case ? (we use Ceph under > >> >> > an > >> >> > Openstack for glance, nova and cinder > >> >> > >> >> We feel like the current defaults are a good mix of features for > >> >> everyday use of non-shared images or non-krbd images. Most > >> >> importantly, all the default features can be dynamically disabled if > >> >> your needs for the image change. > >> >> > >> >> > > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > ceph-users mailing list > >> >> > ceph-users@lists.ceph.com > >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Jason > >> > > >> > > >> > >> > >> > >> -- > >> Jason > > > > > > > > -- > Jason >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
