[ceph-users] Re: inconsistent pg after upgrade nautilus to octopus

2021-12-19 Thread Christian Rohmann 

Hello Tomasz,


I observe a strange accumulation of inconsistencies for an RGW-only 
(+multisite) setup, with errors just like those you reported.
I collected some info and raised a bug ticket:  
https://tracker.ceph.com/issues/53663
Two more inconsistencies have just shown up hours after repairing the 
other, adding to the theory of something really odd going on.




Did you upgrade to Octopus in the end then? Any more issues with such 
inconsistencies on your side Tomasz?




Regards

Christian



On 20/10/2021 10:33, Tomasz Płaza wrote:
As the upgrade process states, rgw are the last one to be upgraded, so 
they are still on nautilus (centos7). Those logs showed up after 
upgrade of the first osd host. It is a multisite setup so I am a 
little afraid of upgrading rgw now.


Etienne:

Sorry for answering in this thread, but somehow I do not get messages 
directed only to ceph-users list. I did "rados list-inconsistent-pg" 
and got many entries like:


{
  "object": {
    "name": ".dir.99a07ed8-2112-429b-9f94-81383220a95b.7104621.23.7",
    "nspace": "",
    "locator": "",
    "snap": "head",
    "version": 82561410
  },
  "errors": [
    "omap_digest_mismatch"
  ],
  "union_shard_errors": [],
  "selected_object_info": {
    "oid": {
  "oid": ".dir.99a07ed8-2112-429b-9f94-81383220a95b.7104621.23.7",
  "key": "",
  "snapid": -2,
  "hash": 3316145293,
  "max": 0,
  "pool": 230,
  "namespace": ""
    },
    "version": "107760'82561410",
    "prior_version": "106468'82554595",
    "last_reqid": "client.392341383.0:2027385771",
    "user_version": 82561410,
    "size": 0,
    "mtime": "2021-10-19T16:32:25.699134+0200",
    "local_mtime": "2021-10-19T16:32:25.699073+0200",
    "lost": 0,
    "flags": [
  "dirty",
  "omap",
  "data_digest"
    ],
    "truncate_seq": 0,
    "truncate_size": 0,
    "data_digest": "0x",
    "omap_digest": "0x",
    "expected_object_size": 0,
    "expected_write_size": 0,
    "alloc_hint_flags": 0,
    "manifest": {
  "type": 0
    },
    "watchers": {}
  },
  "shards": [
    {
  "osd": 56,
  "primary": true,
  "errors": [],
  "size": 0,
  "omap_digest": "0xf4cf0e1c",
  "data_digest": "0x"
    },
    {
  "osd": 58,
  "primary": false,
  "errors": [],
  "size": 0,
  "omap_digest": "0xf4cf0e1c",
  "data_digest": "0x"
    },
    {
  "osd": 62,
  "primary": false,
  "errors": [],
  "size": 0,
  "omap_digest": "0x4bd5703a",
  "data_digest": "0x"
    }
  ]
}


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Luminous: export and migrate rocksdb to dedicated lvm/unit

2021-12-19 Thread Flavio Piccioni 
I tried the same operation in Nautilus, without success

from ceph-bluestore-tool documentation

   -

   if source list has slow volume only - operation isn’t permitted,
   requires explicit allocation via new-db/new-wal command.

so i tried

ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD}
bluefs-bdev-new-db --dev-target /dev/bluesfs_db/db-osd${OSD} then
ceph-bluestore-tool --path /dev/osd1/ --devs-source dev/osd1/block
--dev-target dev/osd1/block.db bluefs-bdev-migrate

(and many others syntax combinations)
but i always get "too many positional options have been specified on the
command line"


Maybe "bluefs-bdev-new-db" is sufficient to have "slow's integrated
rocksdb" migrated?
Regards
Flavio

Il giorno ven 17 dic 2021 alle ore 18:57 Anthony D'Atri <
anthony.da...@gmail.com> ha scritto:

> Or incrementally destroy and redeploy the OSDs, which will be slower and
> entail a lot of data movement.
>
> >
> > Hey Flavio,
> >
> > I think there are no options other then either upgrade the cluster or
> backport the relevant bluefs migration code to Lumnous and make a custom
> build.
> >
> >
> > Thanks,
> >
> > Igor
> >
> > On 12/17/2021 4:43 PM, Flavio Piccioni wrote:
> >> Hi all,
> >> in a Luminous+Bluestore cluster, I would like to migrate rocksdb
> (including
> >> wal) to nvme (lvm).
> >>
> >> (output comes from test env. with minimum sized hdd to test procedures)
> >> ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0
> >> infering bluefs devices from bluestore path
> >> {
> >> "/var/lib/ceph/osd/ceph-0/block": {
> >> "osd_uuid": "399e7751-d791-4493-9f53-caf1650573ed",
> >> "size": 107369988096,
> >> "btime": "2021-12-16 16:24:32.412358",
> >> "description": "main",
> >> "bluefs": "1",
> >> "ceph_fsid": "uuid",
> >> "kv_backend": "rocksdb",
> >> "magic": "ceph osd volume v026",
> >> "mkfs_done": "yes",
> >> "osd_key": "mykey",
> >> "ready": "ready",
> >> "require_osd_release": "\u000e",
> >> "whoami": "0"
> >> }
> >> }
> >> rocksdb and wal are integrated in slowfs, so there is no rock.db o .wal
> >> entry
> >>
> >> In Luminous and Mimic, there is no bluefs-bdev-new-db option for
> >> ceph-bluestore-tool.
> >> How can this dump+migration be archived in old versions?
> >>
> >> Regards
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> > --
> > Igor Fedotov
> > Ceph Lead Developer
> >
> > Looking for help with your Ceph cluster? Contact us at https://croit.io
> >
> > croit GmbH, Freseniusstr. 31h, 81247 Munich
> > CEO: Martin Verges - VAT-ID: DE310638492
> > Com. register: Amtsgericht Munich HRB 231263
> > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] radosgw-admin bucket chown problems

2021-12-19 Thread Amir Malekzadeh 
Hi,

Has anyone had any luck changing a big bucket's owner using "radosgw-admin
bucket chown" command? I have two issues with this command:

1. It has memory issues: As mentioned here
, its memory usage just rises as it
goes. For a bucket with about a million objects, it uses about 20GB of ram.

2. The --marker option doesn't seem to work: Ceph docs
 says:

> bucket chown

Link bucket to specified user and update object ACLs. *Use –marker to
resume if command gets interrupted*.

But when I try using --marker, it just starts over. I posted this one here
, as well.

These issues are seen on 15.2.12, by the way.

Thanks,
Amir
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: bunch of " received unsolicited reservation grant from osd" messages in log

2021-12-19 Thread Ronen Friedman 
On Sat, Dec 18, 2021 at 7:06 PM Ronen Friedman  wrote:

> Hi all,
>
> This was indeed a bug, which I've already fixed in 'master'.
> I'll look for the backporting status tomorrow.
>
> Ronen
>
>
The fix is part of a larger change (which fixes a more severe issue).
Pending (non-trivial) backport.
I'll try to speed this up.

Ronen





> On Fri, Dec 17, 2021 at 1:49 PM Kenneth Waegeman <
> kenneth.waege...@ugent.be> wrote:
>
>> Hi all,
>>
>> I'm also seeing these messages spamming the logs after update from
>> octopus to pacific 16.2.7.
>>
>> Any clue yet what this means?
>>
>> Thanks!!
>>
>> Kenneth
>>
>> On 29/10/2021 22:21, Alexander Y. Fomichev wrote:
>> > Hello.
>> > After upgrading to 'pacific' I found log spammed by messages like this:
>> > ... active+clean]  scrubber pg(46.7aas0) handle_scrub_reserve_grant:
>> > received unsolicited reservation grant from osd 138(1) (0x560e77c51600)
>> >
>> > If I understand it correctly this is exactly what it looks, and this is
>> not
>> > good. Running with debug osd 1/5 don't help much  and google bring me
>> > nothing and I stuck. Could anybody give a hint what's happening or where
>> >   to dig.
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io