On Fri, Aug 17, 2018 at 12:25 PM David Turner <drakonst...@gmail.com> wrote:
> > I'll open a tracker ticket to fix the issue. > > Is there a tracker URL we can follow along with? > It's here: [1] > > On Thu, Aug 16, 2018 at 10:04 PM Glen Baars <g...@onsitecomputers.com.au> > wrote: > >> Thanks for your help 😊 >> >> Kind regards, >> >> *Glen Baars* >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Thursday, 16 August 2018 10:21 PM >> >> >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* ceph-users <ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> On Thu, Aug 16, 2018 at 2:37 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Is there any workaround that you can think of to correctly enable >> journaling on locked images? >> >> >> >> You could add the "rbd journal pool = XYZ" configuration option to the >> ceph.conf on the hosts currently using the images (or use 'rbd image-meta >> set <image-spec> conf_rbd_journal_pool SSDPOOL' on each image), >> restart/live-migrate the affected VMs(?) to pick up the config changes, and >> enable journaling. >> >> >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* ceph-users <ceph-users-boun...@lists.ceph.com> *On Behalf Of *Glen >> Baars >> *Sent:* Tuesday, 14 August 2018 9:36 PM >> *To:* dilla...@redhat.com >> *Cc:* ceph-users <ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> Hello Jason, >> >> >> >> Thanks for your help. Here is the output you asked for also. >> >> >> >> https://pastebin.com/dKH6mpwk >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Tuesday, 14 August 2018 9:33 PM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* ceph-users <ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> On Tue, Aug 14, 2018 at 9:31 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Jason, >> >> >> >> I have now narrowed it down. >> >> >> >> If the image has an exclusive lock – the journal doesn’t go on the >> correct pool. >> >> >> >> OK, that makes sense. If you have an active client on the image holding >> the lock, the request to enable journaling is sent over to that client but >> it's missing all the journal options. I'll open a tracker ticket to fix the >> issue. >> >> >> >> Thanks. >> >> >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Tuesday, 14 August 2018 9:29 PM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* ceph-users <ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> >> >> On Tue, Aug 14, 2018 at 9:19 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Jason, >> >> >> >> I have tried with and without ‘rbd journal pool = rbd’ in the ceph.conf. >> it doesn’t seem to make a difference. >> >> >> >> It should be SSDPOOL, but regardless, I am at a loss as to why it's not >> working for you. You can try appending "--debug-rbd=20" to the end of the >> "rbd feature enable" command and provide the generated logs in a pastebin >> link. >> >> >> >> Also, here is the output: >> >> >> >> rbd image-meta list RBD-HDD/2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> There are 0 metadata on this image. >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Tuesday, 14 August 2018 9:00 PM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* dillaman <dilla...@redhat.com>; ceph-users < >> ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> I tried w/ a rbd CLI from 12.2.7 and I still don't have an issue enabling >> journaling on a different pool: >> >> >> >> $ rbd info rbd/foo >> >> rbd image 'foo': >> >> size 1024 MB in 256 objects >> >> order 22 (4096 kB objects) >> >> block_name_prefix: rbd_data.101e6b8b4567 >> >> format: 2 >> >> features: layering, exclusive-lock, object-map, fast-diff, >> deep-flatten >> >> flags: >> >> create_timestamp: Tue Aug 14 08:51:19 2018 >> >> $ rbd feature enable rbd/foo journaling --journal-pool rbd_ssd >> >> $ rbd journal info --pool rbd --image foo >> >> rbd journal '101e6b8b4567': >> >> header_oid: journal.101e6b8b4567 >> >> object_oid_prefix: journal_data.1.101e6b8b4567. >> >> order: 24 (16384 kB objects) >> >> splay_width: 4 >> >> object_pool: rbd_ssd >> >> >> >> Can you please run "rbd image-meta list <image-spec>" to see if you are >> overwriting any configuration settings? Do you have any client >> configuration overrides in your "/etc/ceph/ceph.conf"? >> >> >> >> On Tue, Aug 14, 2018 at 8:25 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Jason, >> >> >> >> I will also complete testing of a few combinations tomorrow to try and >> isolate the issue now that we can get it to work with a new image. >> >> >> >> The cluster started out at 12.2.3 bluestore so there shouldn’t be any old >> issues from previous versions. >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Tuesday, 14 August 2018 7:43 PM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* dillaman <dilla...@redhat.com>; ceph-users < >> ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> On Tue, Aug 14, 2018 at 4:08 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Jason, >> >> >> >> I can confirm that your tests work on our cluster with a newly created >> image. >> >> >> >> We still can’t get the current images to use a different object pool. Do >> you think that maybe another feature is incompatible with this feature? >> Below is a log of the issue. >> >> >> >> I wouldn't think so. I used master branch for my testing but I'll try >> 12.2.7 just in case it's an issue that's only in the luminous release. >> >> >> >> :~# rbd info RBD_HDD/2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> rbd image '2ef34a96-27e0-4ae7-9888-fd33c38f657a': >> >> size 51200 MB in 12800 objects >> >> order 22 (4096 kB objects) >> >> block_name_prefix: rbd_data.37c8974b0dc51 >> >> format: 2 >> >> features: layering, exclusive-lock, object-map, fast-diff, >> deep-flatten >> >> flags: >> >> create_timestamp: Sat May 5 11:39:07 2018 >> >> >> >> :~# rbd journal info --pool RBD_HDD --image >> 2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> rbd: journaling is not enabled for image >> 2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> >> >> :~# rbd feature enable RBD_HDD/2ef34a96-27e0-4ae7-9888-fd33c38f657a >> journaling --journal-pool RBD_SSD >> >> >> >> :~# rbd journal info --pool RBD_HDD --image >> 2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> rbd journal '37c8974b0dc51': >> >> header_oid: journal.37c8974b0dc51 >> >> object_oid_prefix: journal_data.1.37c8974b0dc51. >> >> order: 24 (16384 kB objects) >> >> splay_width: 4 >> >> ***************<NOTE NO object_pool> **************** >> >> >> >> :~# rbd info RBD_HDD/2ef34a96-27e0-4ae7-9888-fd33c38f657a >> >> rbd image '2ef34a96-27e0-4ae7-9888-fd33c38f657a': >> >> size 51200 MB in 12800 objects >> >> order 22 (4096 kB objects) >> >> block_name_prefix: rbd_data.37c8974b0dc51 >> >> format: 2 >> >> features: layering, exclusive-lock, object-map, fast-diff, >> deep-flatten, journaling >> >> flags: >> >> create_timestamp: Sat May 5 11:39:07 2018 >> >> journal: 37c8974b0dc51 >> >> mirroring state: disabled >> >> >> >> Kind regards, >> >> *Glen Baars* >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Tuesday, 14 August 2018 12:04 AM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* dillaman <dilla...@redhat.com>; ceph-users < >> ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> On Sun, Aug 12, 2018 at 12:13 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Jason, >> >> >> >> Interesting, I used ‘rados ls’ to view the SSDPOOL and can’t see any >> objects. Is this the correct way to view the journal objects? >> >> >> >> You won't see any journal objects in the SSDPOOL until you issue a write: >> >> >> >> $ rbd create --size 1G --image-feature exclusive-lock rbd_hdd/test >> >> $ rbd bench --io-type=write --io-pattern=rand --io-size=4K --io-total=16M >> rbd_hdd/test --rbd-cache=false >> >> bench type write io_size 4096 io_threads 16 bytes 16777216 pattern random >> >> SEC OPS OPS/SEC BYTES/SEC >> >> 1 320 332.01 1359896.98 >> >> 2 736 360.83 1477975.96 >> >> 3 1040 351.17 1438393.57 >> >> 4 1392 350.94 1437437.51 >> >> 5 1744 350.24 1434576.94 >> >> 6 2080 349.82 1432866.06 >> >> 7 2416 341.73 1399731.23 >> >> 8 2784 348.37 1426930.69 >> >> 9 3152 347.40 1422966.67 >> >> 10 3520 356.04 1458356.70 >> >> 11 3920 361.34 1480050.97 >> >> elapsed: 11 ops: 4096 ops/sec: 353.61 bytes/sec: 1448392.06 >> >> $ rbd feature enable rbd_hdd/test journaling --journal-pool rbd_ssd >> >> $ rbd journal info --pool rbd_hdd --image test >> >> rbd journal '10746b8b4567': >> >> header_oid: journal.10746b8b4567 >> >> object_oid_prefix: journal_data.2.10746b8b4567. >> >> order: 24 (16 MiB objects) >> >> splay_width: 4 >> >> object_pool: rbd_ssd >> >> $ rbd bench --io-type=write --io-pattern=rand --io-size=4K --io-total=16M >> rbd_hdd/test --rbd-cache=false >> >> bench type write io_size 4096 io_threads 16 bytes 16777216 pattern random >> >> SEC OPS OPS/SEC BYTES/SEC >> >> 1 240 248.54 1018005.17 >> >> 2 512 263.47 1079154.06 >> >> 3 768 258.74 1059792.10 >> >> 4 1040 258.50 1058812.60 >> >> 5 1312 258.06 1057001.34 >> >> 6 1536 258.21 1057633.14 >> >> 7 1792 253.81 1039604.73 >> >> 8 2032 253.66 1038971.01 >> >> 9 2256 241.41 988800.93 >> >> 10 2480 237.87 974335.65 >> >> 11 2752 239.41 980624.20 >> >> 12 2992 239.61 981440.94 >> >> 13 3200 233.13 954887.84 >> >> 14 3440 237.36 972237.80 >> >> 15 3680 239.47 980853.37 >> >> 16 3920 238.75 977920.70 >> >> elapsed: 16 ops: 4096 ops/sec: 245.04 bytes/sec: 1003692.81 >> >> $ rados -p rbd_ssd ls | grep journal_data.2.10746b8b4567. >> >> journal_data.2.10746b8b4567.3 >> >> journal_data.2.10746b8b4567.0 >> >> journal_data.2.10746b8b4567.2 >> >> journal_data.2.10746b8b4567.1 >> >> >> >> rbd feature enable SLOWPOOL/RBDImage journaling --journal-pool SSDPOOL >> >> The symptoms that we are experiencing is a huge decrease in write speed ( >> 1QD 128K writes from 160MB/s down to 14MB/s ). We see no improvement when >> moving the journal to SSDPOOL ( but we don’t think it is really moving ) >> >> >> >> If you are trying to optimize for 128KiB writes, you might need to tweak >> the "rbd_journal_max_payload_bytes" setting since it currently is defaulted >> to split journal write events into a maximum of 16KiB payload [1] in order >> to optimize the worst-case memory usage of the rbd-mirror daemon for >> environments w/ hundreds or thousands of replicated images. >> >> >> >> Kind regards, >> >> *Glen Baars* >> >> >> >> *From:* Jason Dillaman <jdill...@redhat.com> >> *Sent:* Saturday, 11 August 2018 11:28 PM >> *To:* Glen Baars <g...@onsitecomputers.com.au> >> *Cc:* ceph-users <ceph-users@lists.ceph.com> >> *Subject:* Re: [ceph-users] RBD journal feature >> >> >> >> On Fri, Aug 10, 2018 at 3:01 AM Glen Baars <g...@onsitecomputers.com.au> >> wrote: >> >> Hello Ceph Users, >> >> >> >> I am trying to implement image journals for our RBD images ( required for >> mirroring ) >> >> >> >> rbd feature enable SLOWPOOL/RBDImage journaling --journal-pool SSDPOOL >> >> >> >> When we run the above command we still find the journal on the SLOWPOOL >> and not on the SSDPOOL. We are running 12.2.7 and all bluestore. We have >> also tried the ceph.conf option (rbd journal pool = SSDPOOL ) >> >> Has anyone else gotten this working? >> >> The journal header was on SLOWPOOL or the journal data objects? I would >> expect that the journal metadata header is located on SLOWPOOL but all data >> objects should be created on SSDPOOL as needed. >> >> >> >> Kind regards, >> >> *Glen Baars* >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> [1] https://github.com/ceph/ceph/blob/master/src/common/options.cc#L6600 >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> >> >> -- >> >> Jason >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> >> >> >> -- >> >> Jason >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > [1] http://tracker.ceph.com/issues/26939 -- Jason
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
