[ceph-users] Quincy: Stuck on image permissions
Hello guys, could someone help me with this? We've been long-time CEPH users... runing several Mimic + Pacific CEPH clusters. Dozens of disk per cluster, typically. BUT... now I have this brand new Quincy cluster and I'm not able to give CLIENT (Quincy on Rocky 8) rw access to ONE IMAGE on Quincy cluster (cephadm / Rocky 9). I'm using something what worked for us for ages: rbd auth ls: client.xxx key: ... caps: [mon] profile rbd caps: [osd] allow rwx pool prod object_prefix rbd_data.600d1c6723ae; allow rwx pool prod object_prefix rbd_header.600d1c6723ae; allow rx pool prod object_prefix rbd_id.xxx-data rbd info: rbd image 'xxx-data': size 2 TiB in 524288 objects order 22 (4 MiB objects) snapshot_count: 2 id: 600d1c6723ae block_name_prefix: rbd_data.600d1c6723ae format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: rados ls: rbd_data.600d1c6723ae.0003958d rbd_header.600d1c6723ae rbd_id.xxx-data BUT... it DOES NOT WORK. When I try it to map on client it says: 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::GetMetadataRequest: 0x7f3a1c001f40 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::ImageState: 0x555eff78cfc0 failed to open image: (1) Operation not permitted rbd: error opening image xxx-data: (1) Operation not permitted The mapping and access DOES work when I put "osd allow *" into ceph auth. What is the recommended syntax for Quincy? btw: this use case should be mentioned in the manual I think... Thanks! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1
We're running Quincy cluster on Rocky9... its in the podman and also, you can install ceph-common (Quincy version) from packages. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Extremally need help. Openshift cluster is down :c
Hello Eugen, yes i have
Its from object a
...
debug 2023-02-12T07:12:55.469+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:13:05.453+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:13:15.478+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:13:25.477+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:13:35.445+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:13:45.487+ 7f66af51e700 1 mds.gml-okd-cephfs-a
asok_command: status {prefix=status} (starting...)
...
and the same for object b
...
debug 2023-02-12T07:15:41.496+ 7fdf6281e700 1 mds.gml-okd-cephfs-b
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:15:51.479+ 7fdf6281e700 1 mds.gml-okd-cephfs-b
asok_command: status {prefix=status} (starting...)
debug 2023-02-12T07:16:01.477+ 7fdf6281e700 1 mds.gml-okd-cephfs-b
asok_command: status {prefix=status} (starting...)
...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Quincy: Stuck on image permissions
Hello, looks like I've found it -- THE NAMESPACES :) I love it. Thanks! On 11/02/2023 21:37, hi...@cgi.cz wrote: Hello guys, could someone help me with this? We've been long-time CEPH users... runing several Mimic + Pacific CEPH clusters. Dozens of disk per cluster, typically. BUT... now I have this brand new Quincy cluster and I'm not able to give CLIENT (Quincy on Rocky 8) rw access to ONE IMAGE on Quincy cluster (cephadm / Rocky 9). I'm using something what worked for us for ages: rbd auth ls: client.xxx key: ... caps: [mon] profile rbd caps: [osd] allow rwx pool prod object_prefix rbd_data.600d1c6723ae; allow rwx pool prod object_prefix rbd_header.600d1c6723ae; allow rx pool prod object_prefix rbd_id.xxx-data rbd info: rbd image 'xxx-data': size 2 TiB in 524288 objects order 22 (4 MiB objects) snapshot_count: 2 id: 600d1c6723ae block_name_prefix: rbd_data.600d1c6723ae format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: rados ls: rbd_data.600d1c6723ae.0003958d rbd_header.600d1c6723ae rbd_id.xxx-data BUT... it DOES NOT WORK. When I try it to map on client it says: 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::GetMetadataRequest: 0x7f3a1c001f40 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::ImageState: 0x555eff78cfc0 failed to open image: (1) Operation not permitted rbd: error opening image xxx-data: (1) Operation not permitted The mapping and access DOES work when I put "osd allow *" into ceph auth. What is the recommended syntax for Quincy? btw: this use case should be mentioned in the manual I think... Thanks! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Subject: OSDs added, remapped pgs and objects misplaced cycling up and down
Hi,
ceph-16.2.9
I've added some new osds - some added to existing hosts and some on
newly-commissioned hosts. The new osds were added to the data side of an
existing EC 8.3 pool.
I've been waiting for the system to finish remapping / backfilling for
some time. Originally the number of remapped pgs and objects misplaced was
steadily decreasing. However in the last few days both remapped pgs and
objects misplaced have increased several times. Both then steadily
decrease before jumping up again. See below
for some details on this.
I'm guessing these are associated stats, i.e. the number of objects
misplaced reflects the objects in the remapped pgs so it's not surprising
they're moving in concert.
I thought at first that this cycling might be due to the balancer (which
was enabled), but the cycling is continuing after the balancer was
disabled, e.g.:
$ ts; ceph balancer status
2023-02-09 12:39:14
{
"active": true,
"last_optimize_duration": "0:00:00.000313",
"last_optimize_started": "Thu Feb 9 01:38:55 2023",
"mode": "upmap",
"optimize_result": "Too many objects (0.078537 > 0.05) are misplaced; try
again later",
"plans": []
}
$ ts; ceph balancer off
2023-02-09 13:03:27 ceph balancer off
4 days later, the balancer is still inactive, and "last_optimize_started"
indicates it hasn't run since before it was turned off:
$ ts; ceph balancer status
2023-02-13 10:18:18
{
"active": false,
"last_optimize_duration": "0:00:00.000420",
"last_optimize_started": "Thu Feb 9 02:02:56 2023",
"mode": "upmap",
"optimize_result": "Too many objects (0.078101 > 0.05) are misplaced; try
again later",
"plans": []
}
Since the balancer was turned off the cycling continues: "misplaced" gets
down to 5% and then rapidly increases to 9%, in conjunction with "starting
backfill" messages, before starting to slowly decrease once again. See
below for a log extract.
Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal
consequence of adding OSDs?
Will it eventually settle done itself or do I need to "do something"?
Is there some way of telling how much work remains until it all settles
down?
Cheers,
Chris
--
#
# General health
# scrubs disabled until added OSDs complete backfilling etc.
#
$ ceph -s
cluster:
id: c6618970-0ce0-4cb2-bc9a-dd5f29b62e24
health: HEALTH_WARN
noout,noscrub,nodeep-scrub flag(s) set
5945 pgs not deep-scrubbed in time
5945 pgs not scrubbed in time
5 slow ops, oldest one blocked for 464 sec, mon.k2 has slow ops
services:
mon: 5 daemons, quorum k2,b2,b4,k1,b5 (age 5d)
mgr: b4(active, since 3M), standbys: b5, b2
osd: 82 osds: 82 up (since 2w), 82 in (since 2w); 12 remapped pgs
flags noout,noscrub,nodeep-scrub
data:
pools: 16 pools, 5945 pgs
objects: 70.35M objects, 88 TiB
usage: 186 TiB used, 242 TiB / 428 TiB avail
pgs: 50050192/700705500 objects misplaced (7.143%)
5933 active+clean
10 active+remapped+backfill_wait
2active+remapped+backfilling
io:
client: 2.2 MiB/s rd, 1.8 MiB/s wr, 395 op/s rd, 122 op/s wr
recovery: 24 MiB/s, 24 objects/s
#
# remapped pgs steadily decreasing then started periodically increasing
# (extract from periodic "ceph -s")
#
2023-02-06 00:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 26
remapped pgs
2023-02-06 04:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 25
remapped pgs
2023-02-06 06:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 24
remapped pgs
2023-02-06 09:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 23
remapped pgs
2023-02-06 12:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 22
remapped pgs
2023-02-06 14:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 21
remapped pgs
2023-02-06 14:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 20
remapped pgs
2023-02-06 20:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 19
remapped pgs
2023-02-07 00:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 18
remapped pgs
2023-02-07 01:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 17
remapped pgs
2023-02-07 11:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 15
remapped pgs
2023-02-07 16:30:00osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 13
remapped pgs
2023-02-07 22:30:00osd: 82 osds: 82 up (since 9d), 82 in (since 8d); 11
remapped pgs
2023-02-08 15:30:00osd: 82 osds: 82 up (since 9d), 82 in (since 9d); 9
remapped pgs
2023-02-08 21:30:00osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 15 remapped
pgs <<< increase
2023-02-09 07:30:00osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 13
remapped pgs
2023-02-09 13:03:27 ceph balancer off
2023-02-09 22:30:00osd: 82 osds: 82
[ceph-users] recovery for node disaster
I have a cluster of three nodes, with three replicas per pool on cluster
nodes
-
HOST ADDR LABELS STATUS
apcepfpspsp0101 192.168.114.157 _admin mon
apcepfpspsp0103 192.168.114.158 mon _admin
apcepfpspsp0105 192.168.114.159 mon _admin
3 hosts in cluster
-
# ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"type": 1,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
-
epoch 1033
fsid 9c35e594-2392-11ed-809a-005056ae050c
created 2022-08-24T09:53:36.481866+
modified 2023-02-12T18:57:34.447536+
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 51
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client luminous
require_osd_release quincy
stretch_mode_enabled false
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags
hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
pool 2 'k8s-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 541 lfor 0/0/44
flags hashpspool,selfmanaged_snaps max_bytes 75161927680 stripe_width 0
application rbd
pool 3 'k8s-cephfs_metadata' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 543
lfor 0/0/57 flags hashpspool max_bytes 5368709120 stripe_width 0
pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
pool 4 'k8s-cephfs_data' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 542
lfor 0/0/57 flags hashpspool max_bytes 32212254720 stripe_width 0
application cephfs
---
Is it possible to recover data when two nodes with all physical disks are
lost for any reason?
What is the maximum number of fault tolerance for the cluster?
For this purpose, consider the default settings .
what changes should I make to increase fault tolerance?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Subject: OSDs added, remapped pgs and objects misplaced cycling up and down
This could be the pg autoscaler since you added new OSDs. You can run ceph
osd pool ls detail and check the pg_num and pg_target numbers iirc to
confirm
On Sun, Feb 12, 2023 at 20:24 Chris Dunlop wrote:
> Hi,
>
> ceph-16.2.9
>
> I've added some new osds - some added to existing hosts and some on
> newly-commissioned hosts. The new osds were added to the data side of an
> existing EC 8.3 pool.
>
> I've been waiting for the system to finish remapping / backfilling for
> some time. Originally the number of remapped pgs and objects misplaced was
> steadily decreasing. However in the last few days both remapped pgs and
> objects misplaced have increased several times. Both then steadily
> decrease before jumping up again. See below
> for some details on this.
>
> I'm guessing these are associated stats, i.e. the number of objects
> misplaced reflects the objects in the remapped pgs so it's not surprising
> they're moving in concert.
>
> I thought at first that this cycling might be due to the balancer (which
> was enabled), but the cycling is continuing after the balancer was
> disabled, e.g.:
>
> $ ts; ceph balancer status
> 2023-02-09 12:39:14
> {
> "active": true,
> "last_optimize_duration": "0:00:00.000313",
> "last_optimize_started": "Thu Feb 9 01:38:55 2023",
> "mode": "upmap",
> "optimize_result": "Too many objects (0.078537 > 0.05) are
> misplaced; try again later",
> "plans": []
> }
> $ ts; ceph balancer off
> 2023-02-09 13:03:27 ceph balancer off
>
> 4 days later, the balancer is still inactive, and "last_optimize_started"
> indicates it hasn't run since before it was turned off:
>
> $ ts; ceph balancer status
> 2023-02-13 10:18:18
> {
> "active": false,
> "last_optimize_duration": "0:00:00.000420",
> "last_optimize_started": "Thu Feb 9 02:02:56 2023",
> "mode": "upmap",
> "optimize_result": "Too many objects (0.078101 > 0.05) are
> misplaced; try again later",
> "plans": []
> }
>
> Since the balancer was turned off the cycling continues: "misplaced" gets
> down to 5% and then rapidly increases to 9%, in conjunction with "starting
> backfill" messages, before starting to slowly decrease once again. See
> below for a log extract.
>
> Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal
> consequence of adding OSDs?
>
> Will it eventually settle done itself or do I need to "do something"?
>
> Is there some way of telling how much work remains until it all settles
> down?
>
> Cheers,
>
> Chris
>
>
> --
> #
> # General health
> # scrubs disabled until added OSDs complete backfilling etc.
> #
> $ ceph -s
>cluster:
> id: c6618970-0ce0-4cb2-bc9a-dd5f29b62e24
> health: HEALTH_WARN
> noout,noscrub,nodeep-scrub flag(s) set
> 5945 pgs not deep-scrubbed in time
> 5945 pgs not scrubbed in time
> 5 slow ops, oldest one blocked for 464 sec, mon.k2 has slow
> ops
>
>services:
> mon: 5 daemons, quorum k2,b2,b4,k1,b5 (age 5d)
> mgr: b4(active, since 3M), standbys: b5, b2
> osd: 82 osds: 82 up (since 2w), 82 in (since 2w); 12 remapped pgs
> flags noout,noscrub,nodeep-scrub
>
>data:
> pools: 16 pools, 5945 pgs
> objects: 70.35M objects, 88 TiB
> usage: 186 TiB used, 242 TiB / 428 TiB avail
> pgs: 50050192/700705500 objects misplaced (7.143%)
> 5933 active+clean
> 10 active+remapped+backfill_wait
> 2active+remapped+backfilling
>
>io:
> client: 2.2 MiB/s rd, 1.8 MiB/s wr, 395 op/s rd, 122 op/s wr
> recovery: 24 MiB/s, 24 objects/s
>
> #
> # remapped pgs steadily decreasing then started periodically increasing
> # (extract from periodic "ceph -s")
> #
> 2023-02-06 00:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d);
> 26 remapped pgs
> 2023-02-06 04:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d);
> 25 remapped pgs
> 2023-02-06 06:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d);
> 24 remapped pgs
> 2023-02-06 09:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 6d);
> 23 remapped pgs
> 2023-02-06 12:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d);
> 22 remapped pgs
> 2023-02-06 14:00:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d);
> 21 remapped pgs
> 2023-02-06 14:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d);
> 20 remapped pgs
> 2023-02-06 20:30:00osd: 82 osds: 82 up (since 7d), 82 in (since 7d);
> 19 remapped pgs
> 2023-02-07 00:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 7d);
> 18 remapped pgs
> 2023-02-07 01:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 7d);
> 17 remapped pgs
> 2023-02-07 11:00:00osd: 82 osds: 82 up (since 8d), 82 in (since 8d);
> 15 remapped pgs
> 2023-02-07 16:30:00osd: 82 osds: 82 up (since 8d), 82 in (since 8d);
> 13 remapped pgs
> 2023-02-07 22:
[ceph-users] Re: Subject: OSDs added, remapped pgs and objects misplaced cycling up and down
On Sun, Feb 12, 2023 at 20:24 Chris Dunlop wrote: Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal consequence of adding OSDs? On Sun, Feb 12, 2023 at 10:02:46PM -0800, Alexandre Marangone wrote: This could be the pg autoscaler since you added new OSDs. You can run ceph osd pool ls detail and check the pg_num and pg_target numbers iirc to confirm $ ceph osd pool ls detail ... pgp_num 46 pgp_num_target 128 ... That indeed explains it - thanks! OK, now I need to find out more about the pg autoscaler: https://docs.ceph.com/en/latest/rados/operations/placement-groups/ For starters: $ ceph osd pool autoscale-status | grep -e ^POOL -e ^rbd.ec.data POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK rbd.ec.data61058G 100.0T 1.375208.0T 0.6610 1.0 128 on False Looks like I maybe should have created that pool with the "--bulk" flag, per https://docs.ceph.com/en/latest/rados/operations/placement-groups/ https://docs.ceph.com/en/latest/rados/operations/placement-groups/ -- The autoscaler uses the bulk flag to determine which pool should start out with a full complement of PGs and only scales down when the usage ratio across the pool is not even. However, if the pool doesn’t have the bulk flag, the pool will start out with minimal PGs and only when there is more usage in the pool. I'll wonder if setting "bulk" now will help get stable faster? $ ceph osd pool set bulk true Cheers, Chris ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
