[ceph-users] Re: Storage class usage stats

[Engelmann Florian]

Is there any PR ongoing to add such counters to bucket stats? rados-level is 
not an option if those counters are needd to do, eg.  rating/billing.


From: Casey Bodley 
Sent: Wednesday, September 9, 2020 7:50:12 PM
To: Tobias Urdin
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Storage class usage stats

That's right, radosgw doesn't do accounting per storage class. All you
have to go on is the rados-level pool stats for those storage classes.

On Mon, Sep 7, 2020 at 7:05 AM Tobias Urdin  wrote:
>
> Hello,
>
> Anybody have any feedback or ways they have resolved this issue?
>
> Best regards
> 
> From: Tobias Urdin 
> Sent: Wednesday, August 26, 2020 3:01:49 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] Storage class usage stats
>
> Hello,
>
> I've been trying to understand if there is any way to get usage information 
> based on storage classes for buckets.
>
> Since there is no information available from the "radosgw-admin bucket stats" 
> commands nor any other endpoint I
> tried to browse the source code but couldn't find any references where the 
> storage class would be exposed in such a way.
>
> It also seems that RadosGW today is not saving any counters on amount of 
> objects stored in storage classes when it's
> collecting usage stats, which means there is no such metadata saved for a 
> bucket.
>
>
> I was hoping it was atleast saved but not exposed because then it would have 
> been a easier fix than adding support to count number of objects in storage 
> classes based on operations which would involve a lot of places and mean 
> writing to the bucket metadata on each op :(
>
>
> Is my assumptions correct that there is no way to retrieve such information, 
> meaning there is no way to measure such usage?
>
> If the answer is yes, I assume the only way to get something that could be 
> measured would be to instead have multiple placement
> targets since that is exposed from in bucket info. The bad things would be 
> though that you lose a lot of functionality related to lifecycle
> and moving a single object to another storage class.
>
> Best regards
> Tobias
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD quota per namespace

[Stefan Kooman]

On 9/24/20 16:01, Jason Dillaman wrote:

On Thu, Sep 24, 2020 at 9:53 AM Stefan Kooman  wrote:


On 2020-09-24 14:34, Eugen Block wrote:

Hi *,

I'm curious if this idea [1] of quotas on namespace level for rbd will
be implemented. I couldn't find any existing commands in my lab Octopus
cluster so I guess it's still just an idea, right?

If there's any information I missed please point me to it.


Correct, it's still on the backlog so it hasn't been implemented yet.
Historically it has been up to higher level provisioning tools built
on top of RBD to implement provisioned space quota support.


A use case that might make use of this support would be Ceph CSI. A 
infra provider can set a quota on RBD namespace usage and Ceph CSI user 
can consume that. No need for separate pools anymore.


Would it be OK if I made a tracker for this? Or what is the best 
approach to get this feature evaluated for implementation / prioritization?


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.6 OSD down, out but container running....

[Eugen Block]

Hi,


=== START OF READ SMART DATA SECTION ===
SMART Health Status: FIRMWARE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS
[asc=5d, ascq=64]


this indicates a slowly failing drive. You should contact the vendor  
and replace the drive. This can happen on new drives, too.



Zitat von Marco Pizzolo :


Thanks Hu Weiwen,

These hosts and drives are perhaps 2 months old or so, and this is the
first cluster we build on them so I was not anticipating a drive issue
already.

The smartmontools show:

root@:~# smartctl -H /dev/sdag
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.11.0-38-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FIRMWARE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS
[asc=5d, ascq=64]

Grown defects during certification 
Total blocks reassigned during format 
Total new blocks reassigned 
Power on minutes since format 
root@:~# smartctl -H /dev/sdah
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.11.0-38-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org



On Wed, Oct 27, 2021 at 1:26 PM 胡 玮文  wrote:


Hi Marco, the log lines are truncated. I recommend you to send the logs to
a file rather than copying from terminal:



cephadm logs --name osd.13 > osd.13.log



I see “read stalled” in the log. Just a guess, can you check the kernel
logs and the SMART info to see if there is something wrong with this disk?
Maybe also do a self-test.



从 Windows 版邮件 发送



*发件人: *Marco Pizzolo 
*发送时间: *2021年10月28日 1:17
*收件人: *胡 玮文 
*抄送: *ceph-users 
*主题: *Re: [ceph-users] 16.2.6 OSD down, out but container running



Is there any command or log I can provide a sample from that would help to
pinpoint the issue?  The 119 of 120 OSDs are working correctly by all
accounts, but I am just unable to have the bring the last one fully online.



Thank you,


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Minimal requirements for ceph csi users?

[Burkhard Linke]

Hi,


I'm currently setting up ceph CSI for our kubernetes cluster. What are 
the minimum requirements / capabilities needed for the rbd and cephfs 
users? The current setup is working well with admin privileges, but I 
would like to reduce it to the necessary minimum.



Regards,

Burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: octupus: stall i/o during recovery

[Peter Lieven]

Hi Istvan,

I have not given Octopus another try yet. But as far as I remember Manuel 
figured out the root cause.
Maybe he can give more insights.

Best,
Peter

Am 28.10.21 um 13:07 schrieb Szabo, Istvan (Agoda):

Hi Peter,

Have you figured out what was the issue?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Peter Lieven 
Sent: Thursday, November 26, 2020 11:37 PM
To: ceph-users@ceph.io
Subject: [ceph-users] octupus: stall i/o during recovery

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

I am currently evaluating ceph and stumbled across an odd issue when an osd 
comes back online.
The osd was taken offline, but is still "in" and is brought back online before it is 
marked "out".

As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD 
down and up procedure.
As OSDs I use 8x 960GB SAS SSDs on 4 Nodes interconnected with 2x10GE each. 
Network and SSD seems not to be congested at any time.

 From time to time I see complete stall in the fio benchmark for approx. 10 
seconds while recovery is ongoing.
All recovery parameteres (max_recovery, max_backfill, sleep etc.) do not seem 
to influence it.

While digging deeper I found that the requests are hanging in the "waiting for 
readable" state.

Help how to debug this further would be great. Might it be linked to the new 
feature in Octopus to read from all OSDs and not just the primary?

Thank you,
Peter
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io



--

Mit freundlichen Grüßen

Peter Lieven

...

  KAMP Netzwerkdienste GmbH
  Vestische Str. 89-91 | 46117 Oberhausen
  Tel: +49 (0) 208.89 402-50 | Fax: +49 (0) 208.89 402-40
  p...@kamp.de | http://www.kamp.de

  Geschäftsführer: Heiner Lante | Michael Lante
  Amtsgericht Duisburg | HRB Nr. 12154
  USt-Id-Nr.: DE 120607556

...


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS and OSD Problems with cephadm@rockylinux solved

[Sebastian Wagner]

In case you still have the error messages and additional info, do you
want to create a tracker issue for this?
https://tracker.ceph.com/projects/orchestrator/issues/new . To me this
sounds like a network issue and not like a rockylinux issue.


Am 26.10.21 um 13:17 schrieb Magnus Harlander:
> Hi,
>
> I solved all my problems mentioned earlier. It boiled down
> to a minimal ceph.conf that was created by cephadm without
> network infos. After replacing the minimal config
> for osd and mds daemons in /var/lib/ceph/UUID/*/config
> everything was fine and osd and mds containers came
> up clean and working.
> Without the public_network directive the daemons didn't
> know which public address to use.
>
> This might be due to rockylinux not explicitly supported
> by cephadm, so id does not know how to parse
> 'ip a' output, or some other strange bug. Btw I'm
> using bonding interfaces and a VM bridge for qemu.
>
> Best regards,
>
> Magnus
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm does not find podman objects for osds

[Sebastian Wagner]

Some thoughts:

  * Do you have any error messages form the MDS daemons?

https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files 
  * Do you have any error messages form the OSDs?
  * What do you mean by "osd podman object"?
  * Try downgrading to 3.0.1


Am 25.10.21 um 23:05 schrieb Magnus Harlander:
> Hi,
>
> after converting my 2 node cluster to cephadm I'm in lots of trouble.
>
> - containerized mds are not available in the cluster. I must
>   run mds from systemd to make my fs available.
>
> - osd podman objects are not found after a reboot of one node. I
>   don't want to test it on the second node, because if it looses
>   the podman config as well, my cluster is dead!
>
> Can anybody help me?
>
> Best regards Magnus
>
> Output from cephadm.log:
>
> cephadm ['--image', 'docker.io/ceph/ceph:v15', '--no-container-init', 'ls']
> 2021-10-25 22:47:01,929 DEBUG container_init=False
> 2021-10-25 22:47:01,929 DEBUG Running command: systemctl is-enabled
> ceph-mds@s1
> 2021-10-25 22:47:01,935 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:01,935 DEBUG Running command: systemctl is-active
> ceph-mds@s1
> 2021-10-25 22:47:01,940 DEBUG systemctl: stdout active
> 2021-10-25 22:47:01,940 DEBUG Running command: ceph -v
> 2021-10-25 22:47:02,009 DEBUG ceph: stdout ceph version 15.2.13
> (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
> 2021-10-25 22:47:02,016 DEBUG Running command: systemctl is-enabled
> ceph-osd@1
> 2021-10-25 22:47:02,024 DEBUG systemctl: stdout disabled
> 2021-10-25 22:47:02,024 DEBUG Running command: systemctl is-active
> ceph-osd@1
> 2021-10-25 22:47:02,031 DEBUG systemctl: stdout inactive
> 2021-10-25 22:47:02,031 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
> 2021-10-25 22:47:02,036 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,036 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
> 2021-10-25 22:47:02,041 DEBUG systemctl: stdout active
> 2021-10-25 22:47:02,041 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,068 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,069 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mon.s1
> 2021-10-25 22:47:02,111 DEBUG /bin/podman: stdout
> 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
> 22:45:39.315992691 +0200 CEST,
> 2021-10-25 22:47:02,203 DEBUG Running command: /bin/podman exec
> 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48 ceph -v
> 2021-10-25 22:47:02,356 DEBUG /bin/podman: stdout ceph version 15.2.13
> (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
> 2021-10-25 22:47:02,434 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
> 2021-10-25 22:47:02,440 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,440 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
> 2021-10-25 22:47:02,445 DEBUG systemctl: stdout active
> 2021-10-25 22:47:02,445 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,468 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,470 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mgr.s1
> 2021-10-25 22:47:02,513 DEBUG /bin/podman: stdout
> 2b08ddb6182e14985939e50a00e2306a77c3068a65ed122dab8bd5604c91af65,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
> 22:45:39.535973449 +0200 CEST,
> 2021-10-25 22:47:02,601 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
> 2021-10-25 22:47:02,607 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,607 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
> 2021-10-25 22:47:02,612 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:02,612 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,636 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,637 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0
> 2021-10-25 22:47:02,709 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0"
> 2021-10-25 22:47:02,712 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.2
> 2021-10-25 22:47:02,718 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,718 DEBUG Runn

[ceph-users] Re: OSD spend too much time on "waiting for readable" -> slow ops -> laggy pg -> rgw stop -> worst case osd restart

[Manuel Lausch]

Hello Istvan,

the state "waiting for readable" seems to be related to this read_lease
topic documented here:
https://docs.ceph.com/en/latest/dev/osd_internals/stale_read/

The only parameter to tune around this I know about, is the
"osd_pool_default_read_lease_ratio" which is defaulted 0.8

My clusters complains after 5 seconds for slow ops due to corresponding
configuration. On starting, stopping or restarting OSDs I see slow
ops up to about 15 seconds. In normal clusters this will not be shown
as slow op because this is under the default 32 seconds limit.

We plan to set this parameter to 0.15 to mitigate the issue before we
will upgrade to ceph octopus and later. I also where happy if someone
with deep knowledge could have a look to this issue.


To your cluster. I don't see the reasen whats triggers your cluster to
run in this stage. Did you stop/start some OSDs beforhand? 
You May tell us some more details. 


Manuel




On Thu, 28 Oct 2021 11:19:15 +
"Szabo, Istvan (Agoda)"  wrote:

> Hi,
> 
> I found on the mail thread couple of emails that might be related to
> this issue, but there it came after osd restart or recovery. My
> cluster is on octopus 15.2.14 at the moment.
> 
> When slow ops started to come I can see laggy pgs and slowly the 3
> rados gateway starts to die behind the haproxy and when the slow ops
> gone it will restart, but it causes user interruption.
> 
> Digging deeper on the affected osd slow_ops historical log I can see
> couple of slow ops, where 32 seconds spent on waiting for readable
> event. Is there anything to do with this? To avoid osd restart I'm
> running with osd_op_thread_suicide_timeout=2000
> osd_op_thread_timeout=90 enabled on all osd at the moment.
> 
> Might be I'm using too small amount of pg?
> Data pool is on 4:2 ec host based and have 7 nodes, currently the pg
> number is 128 based on the balancer suggestion.
> 
> An example slow ops:
> 
> {
> "description": "osd_op(client.141400841.0:290235021
> 28.17s0
> 28:e94c28ab:::9213182a-14ba-48ad-bde9-289a1c0c0de8.6034919.1_%2fWHITELABEL-1%2fPAGETPYE-5%2fDEVICE-4%2fLANGUAGE-38%2fSUBTYPE-0%2f148341:head
> [create,setxattr user.rgw.idtag (56) in=14b,setxattr
> user.rgw.tail_tag (56) in=17b,writefull 0~17706,setxattr
> user.rgw.manifest (375) in=17b,setxattr user.rgw.acl (123)
> in=12b,setxattr user.rgw.content_type (10) in=21b,setxattr
> user.rgw.etag (32) in=13b,setxattr
> user.rgw.x-amz-meta-storagetimestamp (40) in=36b,call
> rgw.obj_store_pg_ver in=19b,setxattr user.rgw.source_zone (4) in=20b]
> snapc 0=[] ondisk+write+known_if_redirected e37602)", "initiated_at":
> "2021-10-28T16:52:44.652426+0700", "age": 3258.4773889160001,
> "duration": 32.445993113, "type_data": { "flag_point": "started",
> "client_info": { "client": "client.141400841", "client_addr":
> "10.118.199.3:0/462844935", "tid": 290235021 }, "events":
> [ { "event": "initiated", "time": "2021-10-28T16:52:44.652426+0700",
> "duration": 0 },
> {
> "event": "throttled",
> "time": "2021-10-28T16:52:44.652426+0700",
> "duration": 6.11439996e-05
> },
> {
> "event": "header_read",
> "time": "2021-10-28T16:52:44.652487+0700",
> "duration": 2.2341e-06
> },
> {
> "event": "all_read",
> "time": "2021-10-28T16:52:44.652489+0700",
> "duration": 1.1519e-06
> },
> {
> "event": "dispatched",
> "time": "2021-10-28T16:52:44.652490+0700",
> "duration": 3.146e-06
> },
> {
> "event": "queued_for_pg",
> "time": "2021-10-28T16:52:44.652493+0700",
> "duration": 4.09360002e-05
> },
> {
> "event": "reached_pg",
> "time": "2021-10-28T16:52:44.652534+0700",
> "duration": 1.9236e-05
> },
> {
> "event": "waiting for readable",
> "time": "2021-10-28T16:52:44.652554+0700",
> "duration": 32.43008682103
> },
> {
> "event": "reached_pg",
> "time": "2021-10-28T16:53:17.082640+0700",
> "duration": 0.000104525001
> },
> {
> "event": "started",
> "time": "2021-10-28T16:53:17.082745+0700",
> "duration": 0.0156739191
> },
>

[ceph-users] Re: [Suspicious newsletter] Re: slow ops at restarting OSDs (octopus)

[Manuel Lausch]

Hello Istvan,

as described on the ceph docu page this ratio will be multiplied with
the value from osd_heartbeat_grace which is defaulted to 20 seconds.

The product is 16 seconds which is the lease ticket time. So your
requests can theoretically be blocked up to 16 seconds. For our usecase
this is way to long to accept this while doing normal maintenance of the
hosts like os-updates or something like this.

with 0.2 the lease ticket times are only 4 seconds which is not realy
nice but we could live with this. The smaller the value will get, the
greater the cpu load will get because of renew this lease tickets. 

This issue we could reproduce with a 3 Node 12 OSD Cluster which only
containd a coulple of test objects. 

Until now I didn't find a satisfying solution.

Manuel


On Thu, 28 Oct 2021 11:56:48 +
"Szabo, Istvan (Agoda)"  wrote:

> Running into the same issue, have you got any answer for the side
> effect? Also what's the reason you've set it to 0.2?
> 
> I started to have this issue above 1 billions of objects and still
> suffering with it.
> 
> -Original Message-
> From: Manuel Lausch  
> Sent: Friday, June 11, 2021 6:35 PM
> To: ceph-users@ceph.io
> Subject: [Suspicious newsletter] [ceph-users] Re: slow ops at
> restarting OSDs (octopus)
> 
> Okay, I poked around a bit more and found this document:
> https://docs.ceph.com/en/latest/dev/osd_internals/stale_read/
> 
> I don't understand exactly what it is all about and how it works, and
> what the intetion is behind it. But there is one config option
> mentiond: "osd_pool_default_read_lease_ratio" This is defaulted to
> 0.8. Multiplied with the osd_hearbeat_grace (which is default 20) it
> sets that "read lease" to 16 seconds ?! 
> 
> I set this ratio to 0.2 which leads to 4 seconds lease time. With
> that, the problem is solved. No more slow ops.
> 
> Until now, I thought that this is a problem on huge clusters. But
> with this setting I assumed that this should be a issue with quite
> small cluster as well. So I tested it with a 3 Node 12 OSD SSD
> Cluster on octopus with the same issues.
> 
> I can't believe I am the first one, which have this problem.
> 
> 
> Manuel
> 
> 
> On Thu, 10 Jun 2021 17:45:02 +0200
> Manuel Lausch  wrote:
> 
> > Hi Peter,
> > 
> > your suggestion pointed me to the right spot. 
> > I didn't know about the feature, that ceph will read from replica
> > PGs.
> > 
> > So on. I found two functions in the osd/PrimaryLogPG.cc:
> > "check_laggy" and "check_laggy_requeue". On both is first a check,
> > if the partners have the octopus features. if not, the function is 
> > skipped. This explains the beginning of the problem after about the 
> > half cluster was updated.
> > 
> > To verifiy this, I added "return true" in the first line of the 
> > functions. The issue is gone with it. But I don't know what
> > problems this could trigger. I know, the root cause is not fixed
> > with it. I think I will open a bug ticket with this knowlage.
> > 
> > 
> > osd_op_queue_cutoff is set to high
> > and a icmp rate limiting should not happen
> > 
> > 
> > Thanks
> > Manuel  
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io



-- 
Manuel Lausch

Systemadministrator
Storage Services

1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 |
76135 Karlsruhe | Germany Phone: +49 721 91374-1847
E-Mail: manuel.lau...@1und1.de | Web: www.1und1.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha
Vollmer


Member of United Internet

Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte
Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat
sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie
bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem
bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern,
weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu
verwenden.

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient of this e-mail, you are hereby
notified that saving, distribution or use of the content of this e-mail
in any way is prohibited. If you have received this e-mail in error,
please notify the sender and delete the e-mail.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Storage class usage stats

[Casey Bodley]

On Thu, Oct 28, 2021 at 3:46 AM Engelmann Florian
 wrote:
>
> Is there any PR ongoing to add such counters to bucket stats? rados-level is 
> not an option if those counters are needd to do, eg.  rating/billing.

i'm not aware of any work in progress here. a feature request at
https://tracker.ceph.com/projects/rgw/issues/new (set Tracker=Feature)
is probably a good place to start

there have been some requests to enable user/bucket quotas on a
per-storage-class basis, and that would probably require that buckets
track these per-storage-class stats as well

> 
> From: Casey Bodley 
> Sent: Wednesday, September 9, 2020 7:50:12 PM
> To: Tobias Urdin
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: Storage class usage stats
>
> That's right, radosgw doesn't do accounting per storage class. All you
> have to go on is the rados-level pool stats for those storage classes.
>
> On Mon, Sep 7, 2020 at 7:05 AM Tobias Urdin  wrote:
> >
> > Hello,
> >
> > Anybody have any feedback or ways they have resolved this issue?
> >
> > Best regards
> > 
> > From: Tobias Urdin 
> > Sent: Wednesday, August 26, 2020 3:01:49 PM
> > To: ceph-users@ceph.io
> > Subject: [ceph-users] Storage class usage stats
> >
> > Hello,
> >
> > I've been trying to understand if there is any way to get usage information 
> > based on storage classes for buckets.
> >
> > Since there is no information available from the "radosgw-admin bucket 
> > stats" commands nor any other endpoint I
> > tried to browse the source code but couldn't find any references where the 
> > storage class would be exposed in such a way.
> >
> > It also seems that RadosGW today is not saving any counters on amount of 
> > objects stored in storage classes when it's
> > collecting usage stats, which means there is no such metadata saved for a 
> > bucket.
> >
> >
> > I was hoping it was atleast saved but not exposed because then it would 
> > have been a easier fix than adding support to count number of objects in 
> > storage classes based on operations which would involve a lot of places and 
> > mean writing to the bucket metadata on each op :(
> >
> >
> > Is my assumptions correct that there is no way to retrieve such 
> > information, meaning there is no way to measure such usage?
> >
> > If the answer is yes, I assume the only way to get something that could be 
> > measured would be to instead have multiple placement
> > targets since that is exposed from in bucket info. The bad things would be 
> > though that you lose a lot of functionality related to lifecycle
> > and moving a single object to another storage class.
> >
> > Best regards
> > Tobias
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [IMPORTANT NOTICE] Potential data corruption in Pacific

[Igor Fedotov]

Dear Ceph users.

On behalf of Ceph's developers community I have to inform about a 
recently discovered severe bug which might cause data corruption. The 
issue occurs during OMAP format conversion for clusters upgraded to 
Pacific, new clusters aren't affected. OMAP format conversion's trigger 
is BlueStore repair/quick-fix functionality which might be invoked 
either manually via ceph-bluestore-tool or automatically by OSD if 
'bluestore_fsck_quick_fix_on_mount' is set to true.


Both OSD and MDS daemons are known to be suffering from the issue, 
potentially other ones, e.g. RGW might be affected as well - the major 
symptom is daemon's inability to startup/proceed operating after some 
OSDs have been "repaired".


More details on the bug and its status tracking can be found at: 
https://tracker.ceph.com/issues/53062


We're currently working on the fix which is expected to be available in 
the upcoming v16.2.7 release.


Meanwhile please DO NOT SET bluestore_fsck_quick_fix_on_mount to true 
(please immediately switch it to false if already set) and DO NOT RUN 
ceph-bluestore-tool's repair/quick-fix commands.


Appologies for all the troubles this could cause.

--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Minimal requirements for ceph csi users?

[Konstantin Shalygin]

Hi,

Try to use profile cap, like 'allow profile rbd'


k
Sent from my iPhone

> On 28 Oct 2021, at 12:47, Burkhard Linke 
>  wrote:
> 
> I'm currently setting up ceph CSI for our kubernetes cluster. What are the 
> minimum requirements / capabilities needed for the rbd and cephfs users? The 
> current setup is working well with admin privileges, but I would like to 
> reduce it to the necessary minimum.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [IMPORTANT NOTICE] Potential data corruption in Pacific

[Igor Fedotov]

On 10/28/2021 7:13 PM, Eneko Lacunza wrote:

Hi all,

For those looking for the exact commands, I think they are:

* Check value for bluestore_fsck_quick_fix_on_mount:

ceph config get osd bluestore_fsck_quick_fix_on_mount


* Set bluestore_fsck_quick_fix_on_mount to false:

ceph config set osd bluestore_fsck_quick_fix_on_mount false


We upgraded from Octopus to Pacific 16.2.6, and it seems we got 
"false" as default value (I don't see any false setting in history of 
the servers).


Yes, this parameter has got false as a default setting. So one is quite 
safe with the upgrade by default...





Thanks a lot for the heads up Igor!

Cheers


El 28/10/21 a las 17:37, Igor Fedotov escribió:

Dear Ceph users.

On behalf of Ceph's developers community I have to inform about a 
recently discovered severe bug which might cause data corruption. The 
issue occurs during OMAP format conversion for clusters upgraded to 
Pacific, new clusters aren't affected. OMAP format conversion's 
trigger is BlueStore repair/quick-fix functionality which might be 
invoked either manually via ceph-bluestore-tool or automatically by 
OSD if 'bluestore_fsck_quick_fix_on_mount' is set to true.


Both OSD and MDS daemons are known to be suffering from the issue, 
potentially other ones, e.g. RGW might be affected as well - the 
major symptom is daemon's inability to startup/proceed operating 
after some OSDs have been "repaired".


More details on the bug and its status tracking can be found at: 
https://tracker.ceph.com/issues/53062


We're currently working on the fix which is expected to be available 
in the upcoming v16.2.7 release.


Meanwhile please DO NOT SET bluestore_fsck_quick_fix_on_mount to true 
(please immediately switch it to false if already set) and DO NOT RUN 
ceph-bluestore-tool's repair/quick-fix commands.


Appologies for all the troubles this could cause.



Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Minimal requirements for ceph csi users?

[Burkhard Linke]

Hi,

On 28.10.21 18:10, Konstantin Shalygin wrote:

Hi,

Try to use profile cap, like 'allow profile rbd'



That's fine for csi rbd, thx. Works like a charm so far.


But cephfs is a little different beast. As far as I understand the 
source code, it uses the mgr interface to create subvolumes and 
subvolume groups (e.g. the same API calls used by 'ceph fs subvolume 
create..' and others). The default authx caps generated by 'ceph fs 
authorize' do not seem to be sufficient in this case.



To give an example:

    caps mds = "allow rwps fsname=test"
    caps mon = "allow r fsname=test"
    caps osd = "allow rw tag cephfs data=test"

This is not enough to be used with the cephfs csi driver. The caps are 
fine for accessing the filesystem, e.g. mounting.


Even adding 'mgr = "allow rw"' does not work. If I use the client.admin 
credentials instead everything is fine, but I do not want to expose 
those in kubernetes...



Regards,

Burkhard


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph User + Dev Monthly Meetup

[Neha Ojha]

Hi everyone,

We are kicking off a new monthly meeting for Ceph users to directly
interact with Ceph Developers. The high-level aim of this meeting is
to provide users with a forum to:

- share their experience running Ceph clusters
- provide feedback on Ceph versions they are using
- ask questions and raise concerns on any matters related to Ceph
- provide documentation feedback and suggest improvements

Note that this is not a meeting to discuss design ideas or feature
improvements, we'll continue to use existing CDMs [0] for such
discussions.

The meeting details have been added to the community calendar [1]. The
first meeting will be held on November 18, 2021, 14:00-15:00 UTC and
the agenda is here:
https://pad.ceph.com/p/ceph-user-dev-monthly-minutes

Hope to see you there!

Thanks,
Neha

[0] https://tracker.ceph.com/projects/ceph/wiki/Planning
[1] 
https://calendar.google.com/calendar/u/0/embed?src=9ts9c7lt7u1vic2ijvvqqlf...@group.calendar.google.com

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Ceph] Recovery is very Slow

[Christian Wuerdig]

Yes, just expose each disk as an individual OSD and you'll already be
better off. Depending what type of SSD they are - if they can sustain
high random write IOPS you may even want to consider partitioning each
disk and create 2 OSDs per SSD to make better use of the available IO
capacity.
For all-flash storage CPU utilization is also a factor - generally
fewer cores with a higher clock speed would be preferred over a cpu
with more cores but lower clock speeds in such a setup.


On Thu, 28 Oct 2021 at 21:25, Lokendra Rathour
 wrote:
>
> Hey Janne,
> Thanks for the feedback, we only wanted to have huge space to test more with 
> more data. do you advise some other way to plan this out?
> So I have 15 disks with 1 TB each.  Creating multiple OSD would help or 
> please advise.
>
> thanks,
> Lokendra
>
>
> On Thu, Oct 28, 2021 at 1:52 PM Janne Johansson  wrote:
>>
>> Den tors 28 okt. 2021 kl 10:18 skrev Lokendra Rathour 
>> :
>> >
>> > Hi Christian,
>> > Thanks for the update.
>> > I have 5 SSD on each node i.e. a total of 15 SSD using which I have 
>> > created this RAID 0 Disk, which in Ceph becomes three OSD. Each OSD with 
>> > around 4.4 TB of disk. and in total it is coming around 13.3 TB.
>> > Do you feel local RAID is an issue here? Keeping independent disks can 
>> > help recovery fast or increase the performance? please advice.
>>
>>
>> That is a very poor way to set up ceph storage.
>>
>>
>> --
>> May the most significant bit of your life be positive.
>
>
>
> --
> ~ Lokendra
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Hardware VAR for 500TB cluster with RedHat support.

[accounts@stargate.services]

Hello to the list.

Who is everyone using for a hardware VAR that can build a Ceph cluster 
with RedHat support? Looking to buy a 500TB cluster. 45Drives is high on 
the list, but they don't (yet) offer 24/7 phone support and that is a 
requirement for the business.


We have existing support contracts with RedHat, so I contacted our rep 
and got a meeting going with several people. They told me they will only 
support hardware certified/designed for Ceph, but they have no server 
models they recommend. They said I need to work with another company to 
spect out and buy the hardware from, and they will provide the software 
support.


So they are not making this easy at all.

Who can I contact that can build out and sell me some servers that will 
make RedHat happy to support?


Thanks for any suggestions.






___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 2 OSDs Near Full, Others Under 50%

[Janne Johansson]

Den tors 28 okt. 2021 kl 22:25 skrev Dave Hall :
> Hello,
> I have a Nautilus 14.2.21 cluster with 48 x 12TB OSDs across 6 nodes, with
> 3 new nodes and 24 more OSDs ready to come online.  The bulk of my pools
> are EC 8+2 with a failure domain of OSD.
> Until yesterday one of the original 48 OSDs had failed and been destroyed
> for a few months.  I was finally able to replace this OSD yesterday.
>
> As the backfilling started I noticed that the other 47 OSDs had gotten
> fairly out of balance somehow.  They range from 53 PGs and 46% full to 87
> PGs and 85% full.  I thought the Balancer would be taking care of this, but
> perhaps there is a problem with my settings.
>
> As of this morning I had 1 nearfull OSD.  As of now I have two.  Due to the
> rebuild OSD I still have 70 PGs to get remapped.  On the other hand, the
> rebuild OSD has been assigned 73 PGs, but so far it's only up to 7% full.
>
> From what I've been able to find, it looks like the Balancer won't run
> until after the backfilling is complete.   When I get there I'd like to
> have the right Balancer setting in place to improve the balance before I
> start introducing the new OSDs.
>
> Any advice or insight would be greatly appreciated.  In particular, I
> noticed that my Balancer mode was 'upmap'.  SInce all of my OSDs are the
> same and my crush-map is flat and uniform, recommendations against
> 'crush-compat' mode don't seem to apply.

You should probably have used a tool like
https://github.com/HeinleinSupport/cern-ceph-scripts/blob/master/tools/upmap/upmap-remapped.py
or
https://github.com/digitalocean/pgremapper

The idea behind those are that you set norebalance/nobackfill, then do
your changes (like
adding tons of new OSDs or major weight changes), then run these tools
so that they tell
the cluster via upmap that "the current situation is meant to be like
this" via specific exceptions
placed on each misplaced/remapped PG. This makes the cluster HEALTH_OK again.

After this, you unset nobackfill,norebalance, but no movement starts
since all PGs are where
they are "supposed" to be, given the current upmap exceptions.

Then, as the balancer (in upmap mode) runs, it will figure out that
those PGs actually are placed
on the wrong OSDs, but will do some 8 at a time, by just removing
their upmap exceptions,
and then the cluster moves a few PGs, and when those are done, the
cluster is healthy
again, scrubs can start and so on, then the balancer finds 8 new PGs
to un-misplace and
then it goes like this until all PGs are in their final positions.

Lots less strain on the cluster, no long periods without scrubs, less
operator interventions.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io