[ceph-users] Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0

2020-12-10 Thread Byrne, Thomas (STFC,RAL,SC)
Hi all, Got an odd issue that I'm not sure how to solve on our Nautilus 14.2.9 EC cluster. The primary OSD of an EC 8+3 PG died this morning with a very sad disk (thousands of pending sectors). After the down out interval a new 'up' primary was assigned and the backfill started. Twenty minutes

[ceph-users] Re: Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0

2020-12-10 Thread Byrne, Thomas (STFC,RAL,SC)
777, 217. PG query for reference: https://pastebin.com/ZUUH2mQ6 Cheers, Tom > -Original Message- > From: Byrne, Thomas (STFC,RAL,SC) > Sent: 10 December 2020 18:40 > To: 'ceph-users' > Subject: [ceph-users] Incomplete PG due to primary OSD crashing during EC >

[ceph-users] Re: Incomplete PG due to primary OSD crashing during EC backfill - get_hash_info: Mismatch of total_chunk_size 0

2020-12-11 Thread Byrne, Thomas (STFC,RAL,SC)
r into the logs of the original primary to see if there are anything that looks suspicious around the time of the original write, but for now I'm glad to have a happy cluster in time for the weekend! Cheers, Tom > -Original Message- > From: Byrne, Thomas (STFC,RAL,SC) > Sent

[ceph-users] RGW Bucket notification troubleshooting

2021-01-26 Thread Byrne, Thomas (STFC,RAL,SC)
Hi all, We've been trying to get RGW Bucket notifications working with a RabbitMQ endpoint on our Nautilus 14.2.15 cluster. The gateway host can communicate with the rabbitMQ server just fine, but when RGW tries to send a message to the endpoint, the message never appears in the queue, and we g

[ceph-users] Re: fixing future rctimes

2021-03-23 Thread Byrne, Thomas (STFC,RAL,SC)
Hi Dan, Did you get anywhere with fixing your future rctimes, or understanding why you were getting them in the first place? I think we've run into this problem, future rctimes with no associated future subdir/item. The other similarity is the future rctimes always seem to end in .090, compare

[ceph-users] Re: fixing future rctimes

2021-03-23 Thread Byrne, Thomas (STFC,RAL,SC)
e. Cheers, Tom > -Original Message- > From: Dan van der Ster > Sent: 23 March 2021 11:24 > To: Byrne, Thomas (STFC,RAL,SC) > Cc: ceph-users ; Theofilos Mouratidis > > Subject: Re: [ceph-users] fixing future rctimes > > Hi Tom, > > Teo prepar