[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-19 Thread Ilya Dryomov
On Thu, Sep 15, 2022 at 3:33 PM Arthur Outhenin-Chalandre wrote: > > Hi Ronny, > > > On 15/09/2022 14:32 ronny.lippold wrote: > > hi arthur, some time went ... > > > > i would like to know, if there are some news of your setup. > > do you have replication active running? > > No, there was no chan

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-16 Thread Josef Johansson
Hi, I've added as much logging as I can, still shows nothing. On Fri, 16 Sep 2022 at 21:35, Arthur Outhenin-Chalandre < arthur.outhenin-chalan...@cern.ch> wrote: > Hi Josef, > > > On 16/09/2022 14:15 Josef Johansson wrote: > > Are you guys affected by > > https://tracker.ceph.com/issues/57396 ?

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-16 Thread Arthur Outhenin-Chalandre
Hi Josef, > On 16/09/2022 14:15 Josef Johansson wrote: > Are you guys affected by > https://tracker.ceph.com/issues/57396 ? The issue with journal mode for me was more that the journal replay was slow which made the journal also grows... You should probably inspect your rbd-mirror logs (and if

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-16 Thread Josef Johansson
Hi, Are you guys affected by https://tracker.ceph.com/issues/57396 ? On Fri, 16 Sep 2022 at 09:40, ronny.lippold wrote: > hi and thanks a lot. > good to stay not alone and understand some right :) > > i will also tell, if there is something new. > > > so from my point of view, the only consista

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-16 Thread ronny.lippold
hi and thanks a lot. good to stay not alone and understand some right :) i will also tell, if there is something new. so from my point of view, the only consistant way is to freeze fs or shutdown vm. after that start journal mirroring. so i think, only journal can work. you helped me a lot,

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-15 Thread Arthur Outhenin-Chalandre
Hi Ronny, > On 15/09/2022 14:32 ronny.lippold wrote: > hi arthur, some time went ... > > i would like to know, if there are some news of your setup. > do you have replication active running? No, there was no change at CERN. I am switching jobs as well actually so I won't have much news for you

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-09-15 Thread ronny.lippold
hi arthur, some time went ... i would like to know, if there are some news of your setup. do you have replication active running? we are using actually snapshot based and had last time a move of both clusters. after that, we had some damaged filesystems ind the kvm vms. did you ever had such a

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-24 Thread Arthur Outhenin-Chalandre
Hi Ronny, Not sure what could have cause your outage with journaling TBH :/. Best of luck for the Ceph/Proxmox bug! On 5/23/22 20:09, ronny.lippold wrote: > hi arthur, > > just for information. we had some horrible days ... > > last week, we shut some virtual machines down. > most of them did n

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-23 Thread ronny.lippold
hi arthur, just for information. we had some horrible days ... last week, we shut some virtual machines down. most of them did not came back. timeout qmp socket ... and no kvm console. so, we switched to our rbd-mirror cluster and ... yes, was working, puh. some days later, we tried to insta

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-16 Thread ronny.lippold
Am 2022-05-12 15:29, schrieb Arthur Outhenin-Chalandre: We are going towards mirror snapshots, but we didn't advertise internally so far and we won't enable it on every images; it would only be for new volumes if people want explicitly that feature. So we are probably not going to hit these p

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-12 Thread Arthur Outhenin-Chalandre
On 5/12/22 14:31, ronny.lippold wrote: > many thanks, we will check the slides ... are looking great > > >>> >>> ok, you mean, that the growing came, cause of replication is to slow? >>> strange ... i thought our cluster is not so big ... but ok. >>> so, we cannot use journal ... >>> maybe some e

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-12 Thread ronny.lippold
many thanks, we will check the slides ... are looking great ok, you mean, that the growing came, cause of replication is to slow? strange ... i thought our cluster is not so big ... but ok. so, we cannot use journal ... maybe some else have same result? If you want a bit more details on this

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-12 Thread Arthur Outhenin-Chalandre
On 5/12/22 13:25, ronny.lippold wrote: > hi arthur and thanks for answering, > > > Am 2022-05-12 13:06, schrieb Arthur Outhenin-Chalandre: >> Hi Ronny > >> >> Yes according to my test we were not able to have a good replication >> speed on a single image (I think it was 30Mb/s per image somethin

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-12 Thread ronny.lippold
hi arthur and thanks for answering, Am 2022-05-12 13:06, schrieb Arthur Outhenin-Chalandre: Hi Ronny Yes according to my test we were not able to have a good replication speed on a single image (I think it was 30Mb/s per image something like that). So you have probably a few image that writ

[ceph-users] Re: rbd mirroring - journal growing and snapshot high io load

2022-05-12 Thread Arthur Outhenin-Chalandre
Hi Ronny On 5/12/22 12:47, ronny.lippold wrote: > hi to all here > we tried a lot and now, we need your help ... > > we are using 5 proxmox 7.2-3 server with kernel 5.15.30-2-pve and ceph > 16.2.7.. > per server, we use 9 osd (8x 2tb, 1x8tb both sas ssd, connected via sas > hba) > the second cl