[ceph-users] Re: Correct Migration Workflow Replicated -> Erasure Code
We've solved this off-list (because I already got access to the cluster) For the list: Copying on rados level is possible, but requires to shut down radosgw to get a consistent copy. This wasn't feasible here due to the size and performance. We've instead added a second zone where the placement maps to an EC pool to the zonegroup and it's currently copying over data. We'll then make the second zone master and default and ultimately delete the first one. This allows for a migration without downtime. Another possibility would be using a Transition lifecycle rule, but that's not ideal because it doesn't actually change the bucket. I don't think it would be too complicated to add a native bucket migration mechanism that works similar to "bucket rewrite" (which is intended for something similar but different). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Oct 30, 2019 at 6:46 AM Konstantin Shalygin wrote: > > On 10/29/19 1:40 AM, Mac Wynkoop wrote: > > So, I'm in the process of trying to migrate our rgw.buckets.data pool > > from a replicated rule pool to an erasure coded pool. I've gotten the > > EC pool set up, good EC profile and crush ruleset, pool created > > successfully, but when I go to "rados cppool xxx.rgw.buckets.data > > xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of > > data: > > > > error copying object: (2) No such file or directory > > error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: > > (2) No such file or directory > > > > Is "rados cppool" still the blessed way to do the migration, or has > > something better/not deprecated been developed that I can use? > > rados cppool AFAIK lack of support of copy from replicated to EC. May be > now is wrong. > > Also you can use rados import/export. > > > > k > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12
I am getting these since Nautilus upgrade [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd 911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de 9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da 911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db 911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df 9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd 911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de 9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da 911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db 911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df 9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd 911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de 9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd 911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de 9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd 911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 911d3eef5a00 fail -12 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Lower mem radosgw config?
Hey Dan, We've got three rgws with the following configuration: - We're running 12.2.12 with civit web. - 3 RGW's with haproxy round robin - 32 GiB RAM (handles = 4, thread pool = 512) - We run mon+mgr+rgw on the same hardware. Looking at our grafana dashboards, I don't see us running out of free memory, they hover around 5 GB free (but sometimes lower). >From my quick tests (a fair while ago) I found making more handles eventually lead to OOMs. Currently, we've been getting some "connection reset by peer" (errno 104)... from client reads, which I suspect is due to the low handles setting. We plan to move the rgws onto 125 GiB beefier machines. Cheers, Tom On Mon, Oct 28, 2019 at 6:14 PM Dan van der Ster wrote: > Hi all, > > Does anyone have a good config for lower memory radosgw machines? > We have 16GB VMs and our radosgw's go OOM when we have lots of > parallel clients (e.g. I see around 500 objecter_ops via the rgw > asok). > > Maybe lowering rgw_thread_pool_size from 512 would help? > > (This is running latest luminous). > > Thanks, Dan > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Thomas Bennett Storage Engineer at SARAO ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Correct Migration Workflow Replicated -> Erasure Code
RADOS import/export? Will look into that. I ended up going with Paul's suggestion, and created a separate zone under the main zonegroup with another RGW instance running that zone, and had it sync the data into the Erasure-Coded zone. Made it very easy to do! Thanks, Mac On Wed, Oct 30, 2019 at 12:46 AM Konstantin Shalygin wrote: > On 10/29/19 1:40 AM, Mac Wynkoop wrote: > > So, I'm in the process of trying to migrate our rgw.buckets.data pool > > from a replicated rule pool to an erasure coded pool. I've gotten the > > EC pool set up, good EC profile and crush ruleset, pool created > > successfully, but when I go to "rados cppool xxx.rgw.buckets.data > > xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of > > data: > > > > error copying object: (2) No such file or directory > > error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: > > (2) No such file or directory > > > > Is "rados cppool" still the blessed way to do the migration, or has > > something better/not deprecated been developed that I can use? > > rados cppool AFAIK lack of support of copy from replicated to EC. May be > now is wrong. > > Also you can use rados import/export. > > > > k > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Splitting PGs not happening on Nautilus 14.2.2
This morning I noticed that on a new cluster the number of PGs for the default.rgw.buckets.data pool was way too small (just 8 PGs), but when I try to split the PGs the cluster doesn't do anything: # ceph osd pool set default.rgw.buckets.data pg_num 16 set pool 13 pg_num to 16 It seems to set the pg_num_target/pgp_num_target to 16, but pg_num/pgp_num never increase: # ceph osd dump | grep default.rgw.buckets.data pool 13 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 pg_num_target 16 pgp_num_target 16 autoscale_mode warn last_change 43217 flags hashpspool,creating stripe_width 0 compression_mode aggressive application rgw The clusters has 295 OSDs, so I need to do a bit of splitting today. Any ideas why the splits aren't starting? Thanks, Bryan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Dirlisting hangs with cephfs
Not sure if this is related to the dirlisting issue since the deep-scrubs have always been way behind schedule. But lets see if it has any effect to clear this warning. But it seems i can only deep-scrub 5 pgs at a time. How can i increase this ? On Wed, Oct 30, 2019 at 6:53 AM Lars Täuber wrote: > Hi Kári, > > what about this: > > health: HEALTH_WARN > 854 pgs not deep-scrubbed in time > > > maybe you should > $ ceph –cluster first pg scrub XX.YY > or > $ ceph –cluster first pg deep-scrub XX.YY > all the PGs. > > > Tue, 29 Oct 2019 22:43:28 + > Kári Bertilsson ==> Nathan Fish < > lordci...@gmail.com> : > > I am encountering the dirlist hanging issue on multiple clients and none > of > > them are Ubuntu. > > > > Debian buster running kernel 4.19.0-2-amd64. This one was working fine > > until after ceph was upgraded to nautilus > > > > Proxmox running kernels 5.0.21-1-pve and 5.0.18-1-pve > > > > On Tue, Oct 29, 2019 at 9:04 PM Nathan Fish wrote: > > > > > Ubuntu's 4.15.0-66 has this bug, yes. -65 is safe and -67 will have the > > > fix. > > > > > > On Tue, Oct 29, 2019 at 4:54 PM Patrick Donnelly > > > wrote: > > > > > > > > On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber > wrote: > > > > > > > > > > Hi! > > > > > > > > > > What kind of client (kernel vs. FUSE) do you use? > > > > > I experience a lot of the following problems with the most recent > > > ubuntu 18.04.3 kernel 4.15.0-66-generic : > > > > > kernel: [260144.644232] cache_from_obj: Wrong slab cache. > inode_cache > > > but object is from ceph_inode_info > > > > > > > > > > Other clients with older kernels (e.g. 4.15.0-47-generic) work > without > > > interruption on the same CephFS. > > > > > > > > Sounds like another issue reported on ceph-users: "[ceph-users] > CephFS > > > > kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?" > > > > > > > > Suggest you contact Canonical to get them to fix their kernel or > > > > downgrade your kernel. > > > > > > > > -- > > > > Patrick Donnelly, Ph.D. > > > > He / Him / His > > > > Senior Software Engineer > > > > Red Hat Sunnyvale, CA > > > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > > ___ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > -- > Informationstechnologie > Berlin-Brandenburgische Akademie der Wissenschaften > Jägerstraße 22-23 10117 Berlin > Tel.: +49 30 20370-352 http://www.bbaw.de > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: rgw recovering shards
So, I ended up checking all datalog shards with: radosgw-admin data sync status --shard-id=XY --source-zone=us-east-1 and found one with a few hundred references to a bucket that had been deleted. I ended up shutting down HAProxy on both ends and running radosgw-admin data sync init This seemed to clear out the shards and things are running normally now. radosgw-admin stale instances list didn't find these for some reason. thx Frank On Wed, Oct 30, 2019 at 2:55 AM Konstantin Shalygin wrote: > On 10/29/19 10:56 PM, Frank R wrote: > > oldest incremental change not applied: 2019-10-22 00:24:09.0.720448s > > May be zone period is not the same on both sides? > > > > k > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Splitting PGs not happening on Nautilus 14.2.2
Responding to myself to follow up with what I found. While going over the release notes for 14.2.3/14.2.4 I found this was a known problem that has already been fixed. Upgrading the cluster to 14.2.4 fixed the issue. Bryan > On Oct 30, 2019, at 10:33 AM, Bryan Stillwell wrote: > > This morning I noticed that on a new cluster the number of PGs for the > default.rgw.buckets.data pool was way too small (just 8 PGs), but when I try > to split the PGs the cluster doesn't do anything: > > # ceph osd pool set default.rgw.buckets.data pg_num 16 > set pool 13 pg_num to 16 > > It seems to set the pg_num_target/pgp_num_target to 16, but pg_num/pgp_num > never increase: > > # ceph osd dump | grep default.rgw.buckets.data > pool 13 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 8 pgp_num 8 pg_num_target 16 pgp_num_target 16 > autoscale_mode warn last_change 43217 flags hashpspool,creating stripe_width > 0 compression_mode aggressive application rgw > > The clusters has 295 OSDs, so I need to do a bit of splitting today. Any > ideas why the splits aren't starting? > > Thanks, > Bryan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
