[ceph-users] Bluestore compression - Which algo to choose? Zstd really still that bad?

[Christian Rohmann]

Hey ceph-users,

we've been using the default "snappy" to have Ceph compress data on 
certain pools - namely backups / copies of volumes of a VM environment.

So it's write once, and no random access.
I am now wondering if switching to another algo (there is snappy, zlib, 
lz4, or zstd) would improve the compression ratio (significantly)?


* Does anybody have any real world data on snappy vs. $anyother?

Using zstd is tempting as it's used in various other applications 
(btrfs, MongoDB, ...) for inline-compression with great success.
For Ceph though there is a warning ([1]), about it being not recommended 
in the docs still. But I am wondering if this still stands with e.g. [2] 
merged.
And there was [3] trying to improve the performance, this this reads as 
it only lead to a dead-end and no code changes?



In any case does anybody have any numbers to help with the decision on 
the compression algo?




Regards


Christian


[1] 
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#confval-bluestore_compression_algorithm

[2] https://github.com/ceph/ceph/pull/33790
[3] https://github.com/facebook/zstd/issues/910
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw hang under pressure

[Rok Jaklič]

From
https://swamireddy.wordpress.com/2019/10/23/ceph-sharding-the-rados-gateway-bucket-index/

*Since the index is stored in a single RADOS object, only a single
operation can be done on it at any given time. When the number of objects
increases, the index stored in the RADOS object grows. Since a single index
is handling a large number of objects, and there is a chance the number of
operations also increases, parallelism is not possible which can end up
being a bottleneck. Multiple operations will need to wait in a queue since
a single operation is possible at a time.*

Maybe you know, is this still the case? Article is from 2019.




On Sun, Jun 25, 2023 at 6:22 PM Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com> wrote:

> Hi,
>
> Can you check the read and write latency of your osds?
> Maybe it hangs because it’s waiting for pg’s but maybe the pg are under
> scrub or something else.
> Also with many small objects don’t rely on pg autoscaler, it might not
> tell to increase pg but maybe it should be.
>
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> On 2023. Jun 23., at 19:12, Rok Jaklič  wrote:
>
> Email received from the internet. If in doubt, don't click any link nor
> open any attachment !
> 
>
> We are experiencing something similar (slow GETs responses) when sending 1k
> delete requests for example in ceph v16.2.13.
>
> Rok
>
> On Mon, Jun 12, 2023 at 7:16 PM grin  wrote:
>
> Hello,
>
>
> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
>
> (stable)
>
>
> There is a single (test) radosgw serving plenty of test traffic. When
>
> under heavy req/s ("heavy" in a low sense, about 1k rq/s) it pretty
>
> reliably hangs: low traffic threads seem to work (like handling occasional
>
> PUTs) but GETs are completely nonresponsive, all attention seems to be
>
> spent on futexes.
>
>
> The effect is extremely similar to
>
>
>
> https://ceph-users.ceph.narkive.com/I4uFVzH9/radosgw-civetweb-hangs-once-around-850-established-connections
>
> (subject: Radosgw (civetweb) hangs once around)
>
> except this is quincy so it's beast instead of civetweb. The effect is the
>
> same as described there, except the cluster is way smaller (about 20-40
>
> OSDs).
>
>
> I observed that when I start radosgw -f with debug 20/20 it almost never
>
> hangs, so my guess is some ugly race condition. However I am a bit clueless
>
> how to actually debug it since debugging makes it go away. Debug 1
>
> (default) with -d seems to hang after a while but it's not that simple to
>
> induce, I'm still testing under 4/4.
>
>
> Also I do not see much to configure about beast.
>
>
> As to answer the question in the original (2016) thread:
>
> - Debian stable
>
> - no visible limits issue
>
> - no obvious memory leak observed
>
> - no other visible resource shortage
>
> - strace says everyone's waiting on futexes, about 600-800 threads, apart
>
> from the one serving occasional PUTs
>
> - tcp port doesn't respond.
>
>
> IRC didn't react. ;-)
>
>
> Thanks,
>
> Peter
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Stefan Kooman]

On 6/26/23 08:38, Jorge JP wrote:

Hello,

After deep-scrub my cluster shown this error:

HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
2/77158878 objects degraded (0.000%), 1 pg degraded
[WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
 pg 32.15c has 1 unfound objects
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent
 pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
[49,47], 1 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
(0.000%), 1 pg degraded
 pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
[49,47], 1 unfound


I searching in internet how it solves, but I'm confusing..

Anyone can help me?


Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Jorge JP]

Hello Stefan,

I run this command yesterday but the status not changed. Other pgs with status 
"inconsistent" was repaired after a day, but in this case, not works.

instructing pg 32.15c on osd.49 to repair

Normally, the pg will changed to repair but not.


De: Stefan Kooman 
Enviado: lunes, 26 de junio de 2023 11:27
Para: Jorge JP ; ceph-users@ceph.io 
Asunto: Re: [ceph-users] Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

On 6/26/23 08:38, Jorge JP wrote:
> Hello,
>
> After deep-scrub my cluster shown this error:
>
> HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
> damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
> 2/77158878 objects degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
>  pg 32.15c has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg 
> inconsistent
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
> (0.000%), 1 pg degraded
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
>
>
> I searching in internet how it solves, but I'm confusing..
>
> Anyone can help me?

Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Radogw ignoring HTTP_X_FORWARDED_FOR header

[Yosr Kchaou]

Hello,

We are working on setting up an nginx sidecar container running along a
RadosGW container inside the same kubernetes pod.

We are facing an issue with getting the right value for the header
HTTP_X_FORWARDED_FOR when getting client requests. We need this value to do
the source ip check validation.

Currently, RGW sees that all requests come from 127.0.0.1. So it is still
considering the nginx ip address and not the client who made the request.

Even though, we configured our RGW the following
rgw_remote_addr_param = HTTP_X_FORWARDED_FOR

And from the nginx side, we have the following configuration to set the
headers.

location / {
  proxy_pass http://localhost:7480;

  proxy_set_header X-Real-IP   $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto   $scheme;
  proxy_set_header X-Forwarded-Host$host;
  proxy_set_header X-Forwarded-Port$server_port;

Any idea what is the issue here ?

Thank you in advance,
Yoke
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Frank Schilder]

I don't think pg repair will work. It looks like a 2(1) replicated pool where 
both OSDs seem to have accepted writes while the other was down and now the PG 
can't decide what is the true latest version.

Using size 2 min-size 1 comes with manual labor. As far as I can tell, you will 
need to figure out what files/objects are affected and either update the 
missing copy or delete the object manually.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:34 AM
To: Stefan Kooman; ceph-users@ceph.io
Subject: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Stefan,

I run this command yesterday but the status not changed. Other pgs with status 
"inconsistent" was repaired after a day, but in this case, not works.

instructing pg 32.15c on osd.49 to repair

Normally, the pg will changed to repair but not.


De: Stefan Kooman 
Enviado: lunes, 26 de junio de 2023 11:27
Para: Jorge JP ; ceph-users@ceph.io 
Asunto: Re: [ceph-users] Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

On 6/26/23 08:38, Jorge JP wrote:
> Hello,
>
> After deep-scrub my cluster shown this error:
>
> HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
> damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
> 2/77158878 objects degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
>  pg 32.15c has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg 
> inconsistent
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
> (0.000%), 1 pg degraded
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
>
>
> I searching in internet how it solves, but I'm confusing..
>
> Anyone can help me?

Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Jorge JP]

Hello Frank,

Thank you. I ran the next command: ceph pg 32.15c list_unfound

I located the object but I don't know how solve this problem.

{
"num_missing": 1,
"num_unfound": 1,
"objects": [
{
"oid": {
"oid": "rbd_data.aedf52e8a44410.021f",
"key": "",
"snapid": -2,
"hash": 358991196,
"max": 0,
"pool": 32,
"namespace": ""
},
"need": "49128'125646582",
"have": "0'0",
"flags": "none",
"clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1",
"locations": []
}
],
"more": false


Thank you.


De: Frank Schilder 
Enviado: lunes, 26 de junio de 2023 11:43
Para: Jorge JP ; Stefan Kooman ; 
ceph-users@ceph.io 
Asunto: Re: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

I don't think pg repair will work. It looks like a 2(1) replicated pool where 
both OSDs seem to have accepted writes while the other was down and now the PG 
can't decide what is the true latest version.

Using size 2 min-size 1 comes with manual labor. As far as I can tell, you will 
need to figure out what files/objects are affected and either update the 
missing copy or delete the object manually.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:34 AM
To: Stefan Kooman; ceph-users@ceph.io
Subject: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Stefan,

I run this command yesterday but the status not changed. Other pgs with status 
"inconsistent" was repaired after a day, but in this case, not works.

instructing pg 32.15c on osd.49 to repair

Normally, the pg will changed to repair but not.


De: Stefan Kooman 
Enviado: lunes, 26 de junio de 2023 11:27
Para: Jorge JP ; ceph-users@ceph.io 
Asunto: Re: [ceph-users] Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

On 6/26/23 08:38, Jorge JP wrote:
> Hello,
>
> After deep-scrub my cluster shown this error:
>
> HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
> damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
> 2/77158878 objects degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
>  pg 32.15c has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg 
> inconsistent
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
> (0.000%), 1 pg degraded
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
>
>
> I searching in internet how it solves, but I'm confusing..
>
> Anyone can help me?

Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Frank Schilder]

Hi Jorge,

neither do I. You will need to wait for help on the list or try to figure 
something out with the docs.

Please be patient, a mark-unfound-lost is only needed if everything else has 
been tried and failed. Until then, clients that don't access the broken object 
should work fine.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:56 AM
To: Frank Schilder; Stefan Kooman; ceph-users@ceph.io
Subject: RE: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Frank,

Thank you. I ran the next command: ceph pg 32.15c list_unfound

I located the object but I don't know how solve this problem.

{
"num_missing": 1,
"num_unfound": 1,
"objects": [
{
"oid": {
"oid": "rbd_data.aedf52e8a44410.021f",
"key": "",
"snapid": -2,
"hash": 358991196,
"max": 0,
"pool": 32,
"namespace": ""
},
"need": "49128'125646582",
"have": "0'0",
"flags": "none",
"clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1",
"locations": []
}
],
"more": false


Thank you.


De: Frank Schilder 
Enviado: lunes, 26 de junio de 2023 11:43
Para: Jorge JP ; Stefan Kooman ; 
ceph-users@ceph.io 
Asunto: Re: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

I don't think pg repair will work. It looks like a 2(1) replicated pool where 
both OSDs seem to have accepted writes while the other was down and now the PG 
can't decide what is the true latest version.

Using size 2 min-size 1 comes with manual labor. As far as I can tell, you will 
need to figure out what files/objects are affected and either update the 
missing copy or delete the object manually.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:34 AM
To: Stefan Kooman; ceph-users@ceph.io
Subject: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Stefan,

I run this command yesterday but the status not changed. Other pgs with status 
"inconsistent" was repaired after a day, but in this case, not works.

instructing pg 32.15c on osd.49 to repair

Normally, the pg will changed to repair but not.


De: Stefan Kooman 
Enviado: lunes, 26 de junio de 2023 11:27
Para: Jorge JP ; ceph-users@ceph.io 
Asunto: Re: [ceph-users] Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

On 6/26/23 08:38, Jorge JP wrote:
> Hello,
>
> After deep-scrub my cluster shown this error:
>
> HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
> damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
> 2/77158878 objects degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
>  pg 32.15c has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg 
> inconsistent
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
> (0.000%), 1 pg degraded
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
>
>
> I searching in internet how it solves, but I'm confusing..
>
> Anyone can help me?

Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm and remoto package

[Florian Haas]

Hi Shashi,

I just ran into this myself, and I thought I'd share the 
solution/workaround that I applied.


On 15/05/2023 22:08, Shashi Dahal wrote:

Hi,
I followed this documentation:

https://docs.ceph.com/en/pacific/cephadm/adoption/

This is the error I get when trying to enable cephadm.

ceph mgr module enable cephadm

Error ENOENT: module 'cephadm' reports that it cannot run on the active
manager daemon: loading remoto library:No module named 'remoto' (pass
--force to force enablement)

When I import remoto, it imports just fine.


OS is ubuntu 20.04 focal



As far as I can see, this issue applies to non-containerized Ceph 
Pacific deployments — such as ones orchestrated with ceph-ansible — 
running on Debian or Ubuntu. There is no python3-remoto package on those 
platforms, so you can't install remoto by "regular" installation means 
(that is, apt/apt-get).


It looks to me like this issue was introduced in Pacific, and then went 
away in Quincy because that release dropped remoto and replaced it with 
asyncssh (for which a Debian/Ubuntu package does exist). If you start 
out on Octopus with ceph-ansible and do the Cephadm migration *then*, 
you're apparently fine too, and you can subsequently use Cephadm to 
upgrade to Pacific and Quincy. I think it's just this particular 
combination — (a) run on Debian/Ubuntu, (b) deploy non-containerized, 
*and* (c) start your deployment on Pacific, where Cephadm adoption breaks.


The problem has apparently been known for a while (see 
https://tracker.ceph.com/issues/43415), but the recommendation appears 
to have been "just run mgr on a different OS then", which is frequently 
not a viable option.


I tried (like you did, I assume) to just pip-install remoto, and if I 
opened a Python console and typed "import remoto" it imported just fine, 
but apparently the cephadm mgr module didn't like that.


I've now traced this down to the following line that shows up in the 
ceph-mgr log if you bump "debug mgr" to 10/10:


2023-06-26T10:01:34.799+ 7fb0979ba500 10 mgr[py] Computed sys.path 
'/usr/share/ceph/mgr:/local/lib/python3.8/dist-packages:/lib/python3/dist-packages:/lib/python3.8/dist-packages:lib/python38.zip:/lib/python3.8:/lib/python3.8/lib-dynload'


Note the /local/lib/python3.8/dist-packages path, which does not exist 
on Ubuntu Focal. It's properly /usr/local/lib/python3.8/dist-packages, 
and this is where "pip install", when run as root outside a virtualenv, 
installs packages to.


I think the incorrect sys.path may actually be a build or packaging bug 
in the community packages built for Debian/Ubuntu, but I'm not 100% certain.


At any rate, the combined workaround for this issue, for me, is:

(1) pip install remoto (this installs remoto into 
/usr/local/lib/python3.8/dist-packages)
(2) ln -s /usr/local/lib/python3.8/dist-packages 
/local/lib/python3.8/dist-packages (this makes pip-installed packages 
available to ceph-mgr)

(3) restart all ceph-mgr instances
(4) ceph mgr module enable cephadm

Cheers,
Florian
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph.conf and two different ceph clusters

[garcetto]

good afternoon,
  how can i config ceph.conf file on a generic rbd client to say to use two
different ceph clusters to access different volumes on them?

ceph-cluster-left --> rbd-vol-green
ceph-cluster-right --> rbd-vol-blue

thank you.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph.conf and two different ceph clusters

[Wesley Dillingham]

You need to use the --id and --cluster options of the rbd command and
maintain a .conf file for each cluster.

/etc/ceph/clusterA.conf
/etc/ceph/clusterB.conf

/etc/ceph/clusterA.client.userA.keyring
/etc/ceph/clusterB.client.userB.keyring

now use the rbd commands as such:

rbd --id userA --cluster clusterA

This will cause the client to read the appropriate files (
/etc/ceph/clusterA.client.userA.keyring and  /etc/ceph/clusterA.conf)
The --id and --cluster translate to which file is read for config and
keyring


Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Jun 26, 2023 at 9:15 AM garcetto  wrote:

> good afternoon,
>   how can i config ceph.conf file on a generic rbd client to say to use two
> different ceph clusters to access different volumes on them?
>
> ceph-cluster-left --> rbd-vol-green
> ceph-cluster-right --> rbd-vol-blue
>
> thank you.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Radogw ignoring HTTP_X_FORWARDED_FOR header

[Christian Rohmann]

Hello Yosr,

On 26/06/2023 11:41, Yosr Kchaou wrote:

We are facing an issue with getting the right value for the header
HTTP_X_FORWARDED_FOR when getting client requests. We need this value to do
the source ip check validation.

[...]

Currently, RGW sees that all requests come from 127.0.0.1. So it is still
considering the nginx ip address and not the client who made the request.
May I point you to my recent post to this ML about this very question: 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/IKGLAROSVWHSRZQSYTLLHVRWFPOLBEGL/


I am still planning to reproduce this issue with simple examples and 
headers set manually via e.g. curl to rule out anything stupid I might 
have misconfigured in my case. I just did not find the time yet.


But did you sniff any traffic to the backend or verified how the headers 
look like in your case? Any debug logging "debug rgw = 20" where you can 
see what RGW things of the incoming request?
Did you test with S3 bucket policies or how did you come to the 
conclusion that RGW is not using the X_FORWARDED_FOR header? Or what is 
your indication that things are not working as expected?


From what I can see, the rgw client log does NOT print the external IP 
from the header, but the source IP of the incoming TCP connection:


    2023-06-26T11:14:37.070+ 7f0389e0b700  1 beast: 0x7f051c776660: 
192.168.1.1 - someid [26/Jun/2023:11:14:36.990 +] "PUT 
/bucket/object HTTP/1.1" 200 43248 - "aws-sdk-go/1.27.0 (go1.16.15; 
linux; amd64) S3Manager" - latency=0.07469s



while the rgw ops log does indeed print the remote_address in remote_addr:

{"bucket":"bucket","time":"2023-06-26T11:16:08.721465Z","time_local":"2023-06-26T11:16:08.721465+","remote_addr":"xxx.xxx.xxx.xxx","user":"someuser","operation":"put_obj","uri":"PUT 
/bucket/object 
HTTP/1.1","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":64413,"object_size":64413,"total_time":155,"user_agent":"aws-sdk-go/1.27.0 
(go1.16.15; linux; amd64) 
S3Manager","referrer":"","trans_id":"REDACTED","authentication_type":"Keystone","access_key_id":"REDACTED","temp_url":false}



So in my case it's not that RGW does not receive and logs this info, but 
more about it not applying this in a bucket policy (as far as my 
analysis of the issue goes).




Regards


Christian


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs - unable to create new subvolume

[karon karon]

hi
one last clarification: if I create a new FS i can create subvolumes, i
would like to be able to correct the existing FS thank you for your help.


Le ven. 23 juin 2023 à 10:55, karon karon  a écrit :

> Hello,
>
> I recently use cephfs in version 17.2.6
> I have a pool named "*data*" and a fs "*kube*"
> it was working fine until a few days ago, now i can no longer create a new
> subvolume*, *it gives me the following error:
>
> Error EINVAL: invalid value specified for ceph.dir.subvolume
>>
>
> here is the command used:
>
> ceph fs subvolume create kube newcsivol --pool_layout data
>>
>
> from what I understand it seems that it creates the subvolume but
> immediately puts it in the trash !? here is the log :
>
> 2023-06-23T08:30:53.307+ 7f2b929d2700  0 log_channel(audit) log [DBG]
>> : from='client.86289 -' entity='client.admin' cmd=[{"prefix": "fs subvolume
>> create", "vol_name": "kube", "sub_name": "newcsivol", "group_name": "csi",
>> "pool_layout": "data", "target": ["mon-mgr", ""]}]: dispatch
>> 2023-06-23T08:30:53.307+ 7f2b8a1d1700  0 [volumes INFO
>> volumes.module] Starting _cmd_fs_subvolume_create(group_name:csi,
>> pool_layout:data, prefix:fs subvolume create, sub_name:newcsivol,
>> target:['mon-mgr', ''], vol_name:kube) < ""
>> 2023-06-23T08:30:53.327+ 7f2b8a1d1700  0 [volumes INFO
>> volumes.fs.operations.versions.subvolume_v2] cleaning up subvolume with
>> path: newcsivol
>> 2023-06-23T08:30:53.331+ 7f2b8a1d1700  0 [volumes INFO
>> volumes.fs.operations.versions.subvolume_base] subvolume path
>> 'b'/volumes/csi/newcsivol'' moved to trashcan
>> 2023-06-23T08:30:53.331+ 7f2b8a1d1700  0 [volumes INFO
>> volumes.fs.async_job] queuing job for volume 'kube'
>> 2023-06-23T08:30:53.335+ 7f2b8a1d1700  0 [volumes INFO
>> volumes.module] Finishing _cmd_fs_subvolume_create(group_name:csi,
>> pool_layout:data, prefix:fs subvolume create, sub_name:newcsivol,
>> target:['mon-mgr', ''], vol_name:kube) < ""
>> 2023-06-23T08:30:53.335+ 7f2b8a1d1700 -1 mgr.server reply reply (22)
>> Invalid argument invalid value specified for ceph.dir.subvolume
>> 2023-06-23T08:30:53.339+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.339+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.339+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.339+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.339+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.363+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.363+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.363+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.363+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.363+ 7f2b461bf700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.383+ 7f2b479c2700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.383+ 7f2b479c2700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.383+ 7f2b479c2700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.383+ 7f2b479c2700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.383+ 7f2b479c2700 -1 client.0 error registering
>> admin socket command: (17) File exists
>> 2023-06-23T08:30:53.507+ 7f2b3ff33700  0 [prometheus INFO
>> cherrypy.access.139824530773776] 192.168.240.231 - - [23/Jun/2023:08:30:53]
>> "GET /metrics HTTP/1.1" 200 194558 "" "Prometheus/2.33.4"
>> 2023-06-23T08:30:54.219+ 7f2b3ddaf700  0 [dashboard INFO request] [
>> 172.29.2.142:33040] [GET] [200] [0.003s] [admin] [22.0B]
>> /api/prometheus/notifications
>> 2023-06-23T08:30:54.223+ 7f2b929d2700  0 log_channel(audit) log [DBG]
>> : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format":
>> "json"}]: dispatch
>> 2023-06-23T08:30:54.227+ 7f2b3a5a8700  0 [dashboard INFO request] [
>> 172.29.2.142:49348] [GET] [200] [0.019s] [admin] [22.0B] /api/prometheus
>> 2023-06-23T08:30:54.227+ 7f2b929d2700  0 log_channel(audit) log [DBG]
>> : from='mon.0 -' entity='mon.' cmd=[{"prefix": "balancer status", "format":
>> "json"}]: dispatch
>> 2023-06-23T08:30:54.231+ 7f2b3d5ae700  0 [dashboard INFO request] [
>> 172.29.2.142:39414] [GET] [200] [0.022s] [admin] [9.3K]
>> /api/prometheus/rules
>> 2023-06-23T08:30:54.275+ 7f2ba39d4700  0 log_channel(cluster) log
>> [DBG] : pgmap v211648

[ceph-users] RGW multisite logs (data, md, bilog) not being trimmed automatically?

[Christian Rohmann]

Hey ceph-users,

I am running two (now) Quincy clusters doing RGW multi-site replication 
with only one actually being written to by clients.

The other site is intended simply as a remote copy.

On the primary cluster I am observing an ever growing (objects and 
bytes) "sitea.rgw.log" pool, not so on the remote "siteb.rgw.log" which 
is only 300MiB and around 15k objects with no growth.
Metrics show that the growth of pool on primary is linear for at least 6 
months, so not sudden spikes or anything. Also sync status appears to be 
totally happy.

There are also no warnings in regards to large OMAPs or anything similar.

I was under the impression that RGW will trim its three logs (md, bi, 
data) automatically and only keep data that has not yet been replicated 
by the other zonegroup members?
The config option "ceph config get mgr rgw_sync_log_trim_interval" is 
set to 1200, so 20 Minutes.


So I am wondering if there might be some inconsistency or how I can best 
analyze what the cause for the accumulation of log data is?
There are older questions on the ML, such as [1], but there was not 
really a solution or root cause identified.


I know there is manual trimming, but I'd rather want to analyze the 
current situation and figure out what the cause for the lack of 
auto-trimming is.



  * Do I need to go through all buckets and count logs and look at 
their timestamps? Which queries do make sense here?
  * Is there usually any logging of the log trimming activity that I 
should expect? Or that might indicate why trimming does not happen?



Regards

Christian


[1] 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/WZCFOAMLWV3XCGJ3TVLHGMJFVYNZNKLD/




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg inconsistent

[Frank Schilder]

Just in case, maybe this blog post contains some useful hints:
https://blog.noc.grnet.gr/2016/10/18/surviving-a-ceph-cluster-outage-the-hard-way/

Its on a rather old ceph version, but the operations with objects might still 
be relevant. It requires that at least 1 OSD has a valid copy though.

You should try to find out which file/image this object belongs to from the 
user's perspective. If you have a backup/snapshot, you could mark the object as 
lost and restore a copy of the file/image from backup/snapshot. That's what 
others did in this situation.

You need to search this list for how to find that information. I believe there 
was something with ceph-encoder and low-level rados commands. Search for 
recovery_unfound and "und=found object", there should be many posts.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Frank Schilder 
Sent: Monday, June 26, 2023 12:18 PM
To: Jorge JP; Stefan Kooman; ceph-users@ceph.io
Subject: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hi Jorge,

neither do I. You will need to wait for help on the list or try to figure 
something out with the docs.

Please be patient, a mark-unfound-lost is only needed if everything else has 
been tried and failed. Until then, clients that don't access the broken object 
should work fine.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:56 AM
To: Frank Schilder; Stefan Kooman; ceph-users@ceph.io
Subject: RE: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Frank,

Thank you. I ran the next command: ceph pg 32.15c list_unfound

I located the object but I don't know how solve this problem.

{
"num_missing": 1,
"num_unfound": 1,
"objects": [
{
"oid": {
"oid": "rbd_data.aedf52e8a44410.021f",
"key": "",
"snapid": -2,
"hash": 358991196,
"max": 0,
"pool": 32,
"namespace": ""
},
"need": "49128'125646582",
"have": "0'0",
"flags": "none",
"clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1",
"locations": []
}
],
"more": false


Thank you.


De: Frank Schilder 
Enviado: lunes, 26 de junio de 2023 11:43
Para: Jorge JP ; Stefan Kooman ; 
ceph-users@ceph.io 
Asunto: Re: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

I don't think pg repair will work. It looks like a 2(1) replicated pool where 
both OSDs seem to have accepted writes while the other was down and now the PG 
can't decide what is the true latest version.

Using size 2 min-size 1 comes with manual labor. As far as I can tell, you will 
need to figure out what files/objects are affected and either update the 
missing copy or delete the object manually.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Jorge JP 
Sent: Monday, June 26, 2023 11:34 AM
To: Stefan Kooman; ceph-users@ceph.io
Subject: [ceph-users] Re: Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

Hello Stefan,

I run this command yesterday but the status not changed. Other pgs with status 
"inconsistent" was repaired after a day, but in this case, not works.

instructing pg 32.15c on osd.49 to repair

Normally, the pg will changed to repair but not.


De: Stefan Kooman 
Enviado: lunes, 26 de junio de 2023 11:27
Para: Jorge JP ; ceph-users@ceph.io 
Asunto: Re: [ceph-users] Possible data damage: 1 pg recovery_unfound, 1 pg 
inconsistent

On 6/26/23 08:38, Jorge JP wrote:
> Hello,
>
> After deep-scrub my cluster shown this error:
>
> HEALTH_ERR 1/38578006 objects unfound (0.000%); 1 scrub errors; Possible data 
> damage: 1 pg recovery_unfound, 1 pg inconsistent; Degraded data redundancy: 
> 2/77158878 objects degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/38578006 objects unfound (0.000%)
>  pg 32.15c has 1 unfound objects
> [ERR] OSD_SCRUB_ERRORS: 1 scrub errors
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound, 1 pg 
> inconsistent
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 2/77158878 objects degraded 
> (0.000%), 1 pg degraded
>  pg 32.15c is active+recovery_unfound+degraded+inconsistent, acting 
> [49,47], 1 unfound
>
>
> I searching in internet how it solves, but I'm confusing..
>
> Anyone can help me?

Does "ceph pg repair 32.15c" work for you?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe 

[ceph-users] Re: Radogw ignoring HTTP_X_FORWARDED_FOR header

[yosr . kchaou96]

Hello Christian,

Thanks for your reply.

So running rgw in debug mode shows the correct value of the header 
HTTP_X_FORWARDED_FOR. And my test scenario is trying to execute a GET request 
on a bucket on which policies have been applied. So it should accept requests 
only from a specific ip. And now I am getting 403 Access denied which makes 
sense since, the ip the rgw is evaluating, is the proxy ip and not the 
originator ip. 

Yosr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Radogw ignoring HTTP_X_FORWARDED_FOR header

[yosr . kchaou96]

Hello,

Issue has been found. 

RGW was not supposed to show the originator's ip in the logs. And my test 
scenario was not correct. My client was missing some permissions, that's why I 
get access denied.

Yosr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io