[ceph-users] After ceph rename, radosgw cannot read files via S3 API

2019-11-19 Thread Michal Číla
Hi, after renaming our Ceph Luminous cluster (which we use with RGW), we're unable to get objects stored in RGW via S3 API (classic HTTP get works). We were migrating cluster from one datacenter to another and by doing so, we needed to completely rename hostnames of nodes and also change IP a

Re: [ceph-users] RGW performance with low object sizes

2019-11-19 Thread Christian
Hi, I used https://github.com/dvassallo/s3-benchmark to measure some > performance values for the rgws and got some unexpected results. > Everything above 64K has excellent performance but below it drops down to > a fraction of the speed and responsiveness resulting in even 256K objects > being fa

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread shubjero
Florian, Thanks for posting about this issue. This is something that we have been experiencing (stale exclusive locks) with our OpenStack and Ceph cloud more frequently as our datacentre has had some reliability issues recently with power and cooling causing several unexpected shutdowns. At this

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Jason Dillaman
On Tue, Nov 19, 2019 at 1:51 PM shubjero wrote: > > Florian, > > Thanks for posting about this issue. This is something that we have > been experiencing (stale exclusive locks) with our OpenStack and Ceph > cloud more frequently as our datacentre has had some reliability > issues recently with pow

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Florian Haas
On 19/11/2019 20:03, Jason Dillaman wrote: > On Tue, Nov 19, 2019 at 1:51 PM shubjero wrote: >> >> Florian, >> >> Thanks for posting about this issue. This is something that we have >> been experiencing (stale exclusive locks) with our OpenStack and Ceph >> cloud more frequently as our datacentre

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Jason Dillaman
On Tue, Nov 19, 2019 at 2:49 PM Florian Haas wrote: > > On 19/11/2019 20:03, Jason Dillaman wrote: > > On Tue, Nov 19, 2019 at 1:51 PM shubjero wrote: > >> > >> Florian, > >> > >> Thanks for posting about this issue. This is something that we have > >> been experiencing (stale exclusive locks) wi

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Florian Haas
On 19/11/2019 21:32, Jason Dillaman wrote: >> What, exactly, is the "reasonably configured hypervisor" here, in other >> words, what is it that grabs and releases this lock? It's evidently not >> Nova that does this, but is it libvirt, or Qemu/KVM, and if so, what >> magic in there makes this happe

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Jason Dillaman
On Tue, Nov 19, 2019 at 4:09 PM Florian Haas wrote: > > On 19/11/2019 21:32, Jason Dillaman wrote: > >> What, exactly, is the "reasonably configured hypervisor" here, in other > >> words, what is it that grabs and releases this lock? It's evidently not > >> Nova that does this, but is it libvirt,

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Florian Haas
On 19/11/2019 22:19, Jason Dillaman wrote: > On Tue, Nov 19, 2019 at 4:09 PM Florian Haas wrote: >> >> On 19/11/2019 21:32, Jason Dillaman wrote: What, exactly, is the "reasonably configured hypervisor" here, in other words, what is it that grabs and releases this lock? It's evidently no

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Jason Dillaman
On Tue, Nov 19, 2019 at 4:31 PM Florian Haas wrote: > > On 19/11/2019 22:19, Jason Dillaman wrote: > > On Tue, Nov 19, 2019 at 4:09 PM Florian Haas wrote: > >> > >> On 19/11/2019 21:32, Jason Dillaman wrote: > What, exactly, is the "reasonably configured hypervisor" here, in other > wor

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Florian Haas
On 19/11/2019 22:34, Jason Dillaman wrote: >> Oh totally, I wasn't arguing it was a bad idea for it to do what it >> does! I just got confused by the fact that our mon logs showed what >> looked like a (failed) attempt to blacklist an entire client IP address. > > There should have been an associa

Re: [ceph-users] Global power failure, OpenStack Nova/libvirt/KVM, and Ceph RBD locks

2019-11-19 Thread Jason Dillaman
On Tue, Nov 19, 2019 at 4:42 PM Florian Haas wrote: > > On 19/11/2019 22:34, Jason Dillaman wrote: > >> Oh totally, I wasn't arguing it was a bad idea for it to do what it > >> does! I just got confused by the fact that our mon logs showed what > >> looked like a (failed) attempt to blacklist an e