[2] and edit[3] user with the API.
[1] https://docs.ceph.com/en/reef/mgr/ceph_api/#introduction
[2] https://docs.ceph.com/en/reef/mgr/ceph_api/#post--api-cluster-user
[3] https://docs.ceph.com/en/reef/mgr/ceph_api/#put--api-cluster-user
--
Kai Stian Olstad
000Old_age Always -
37 (Min/Max 28/42)
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On Tue, Sep 17, 2024 at 08:48:11PM -0400, Anthony D'Atri wrote:
Were all three in the same failure domain?
No they were all in different failure domain.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe se
After switching to WPQ and had run deep-scrub on all PG the result was 698
corrupted objects that Ceph could not fix.
So no, I would not say Mclock i production ready.
We have set all out cluster to WPQ.
--
Kai Stian Olstad
___
ceph-users mailing list --
On Fri, Jul 26, 2024 at 04:18:05PM +0200, Iztok Gregori wrote:
On 26/07/24 12:35, Kai Stian Olstad wrote:
On Tue, Jul 23, 2024 at 08:24:21AM +0200, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are
non way to specify an id during the OSD creation?
You
On Tue, Jul 23, 2024 at 08:24:21AM +0200, Iztok Gregori wrote:
Am I missing something obvious or with Ceph orchestrator there are non
way to specify an id during the OSD creation?
You can use osd_id_claims.
This command is for replacing a HDD in hybrid osd.344 and reuse the block.db
device on
On Thu, Jul 18, 2024 at 10:49:02AM +, Eugen Block wrote:
And after restarting the daemon, it seems to work. So my question is,
how do you deal with per-host certificates and rgw? Any comments are
appreciated.
By not dealing with it, sort of.
Since we run our own CA, so I create one certifi
sages. 99% are bogus, but I want to be careful wrt baby and
bathwater.
Will the archive[1] also be clean up?
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubs
sd_recovery_max_active is default to 3.
osd_recovery_op_priority is default to 3, a lower number increases the
priority for recovery.
All of them can be runtime adjusted.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscr
,
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iamuser/"
]
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::*"
}
]
}
And then run
s3cmd setpolicy s3-policy.json s3://
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On Mon, Mar 25, 2024 at 10:58:24PM +0100, Kai Stian Olstad wrote:
On Mon, Mar 25, 2024 at 09:28:01PM +0100, Torkil Svensgaard wrote:
My tally came to 412 out of 539 OSDs showing up in a blocked_by list
and that is about every OSD with data prior to adding ~100 empty
OSDs. How 400 read targets
o the new OSD but also between the existing OSD, but
they are busy with backfilling so they have no free backfill reservation.
I do recommend this document by Joshua Baergen at Digital Ocean that explains
backfilling and the problem with it and there solution, a tool called
pgremapper.
--
Kai
On Sat, Mar 23, 2024 at 12:09:29PM +0100, Torkil Svensgaard wrote:
The other output is too big for pastebin and I'm not familiar with
paste services, any suggestion for a preferred way to share such
output?
You can attached files to the mail here on the list.
--
Kai Stian O
d
was ignored because of the 500 limit.
So only the SSD got the osd_mclock_max_capacity_iops_ssd set.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
pdate osd_mclock_max_capacity_iops_{hdd,ssd} every time
the OSD is started.
If you check the OSD log you'll see it does the bench.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
7]
So maybe just stop that one, wait for recovery and the run deep-scrub to
check if things look better.
But would it then use corrupted data on osd 269 to rebuild.
-
Kai Stian Olstad
On 26.02.2024 10:19, Eugen Block wrote:
Hi,
I think your approach makes sense. But I'm wondering if m
7]
So maybe just stop that one, wait for recovery and the run deep-scrub to
check if things look better.
But would it then use corrupted data on osd 269 to rebuild.
-
Kai Stian Olstad
On 26.02.2024 10:19, Eugen Block wrote:
Hi,
I think your approach makes sense. But I'm wondering if m
since they are now down from the remaining
shards
s1(osd.297), s3(osd.276), s4(osd.136) and s5(osd.197)
_Any_ comment is highly appreciated.
-
Kai Stian Olstad
On 21.02.2024 13:27, Kai Stian Olstad wrote:
Hi,
Short summary
PG 404.bc is an EC 4+2 where s0 and s2 report hash mismtach f
.
- I am asking because I need to use a local registry to deploy
those
images.
You need to check the cephadm source for the version you would like to
use
https://github.com/ceph/ceph/blob/v17.2.7/src/cephadm/cephadm#L46
--
Kai Stian Olstad
_status=500 latency=0.02568s
==
Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: beast: 0x7f94b744d660:
110.2.0.46 - test1 [21/Feb/2024:08:27:06.021 +] "GET
/benchfiles/2021-11-08T19%3A43%3A50%2C145489260%2B00%3A00 HTTP/1.1" 500
226 - - - latency=0.020000568s
ase, EC is 4+2 jerasure blaum_roth and the HDD is hybrid
where DB is on SSD shared by 5 HDD.
And in your cases?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
rgw/
I can find some documentation on octupus
(https://docs.ceph.com/en/octopus/install/ceph-deploy/install-ceph-gateway/)
ceph-deploy is deprecated
https://docs.ceph.com/en/reef/install/
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@
hread/IPHBE3DLW5ABCZHSNYOBUBSI3TLWVD22/#OE3QXLAJIY6NU7PNMGHP47UK2CBZJPUG
-
Kai Stian Olstad
On Tue, Feb 06, 2024 at 06:35:26AM -, LeonGao wrote:
Hi community
We have a new Ceph cluster deployment with 100 nodes. When we are draining an
OSD host from the cluster, we see a small amount of PGs that c
On 31.01.2024 09:38, garcetto wrote:
good morning,
how can i install latest dev release using cephadm?
Have you looked at this page?
https://docs.ceph.com/en/latest/install/containers/#development-builds
--
Kai Stian Olstad
___
ceph-users mailing
he situation
in the test cluster.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
nk you Wes, switching to wpq and restart the OSDs fixed it for me
too.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Hi,
This is a cluster running 17.2.7 upgraded from 16.2.6 on the 15 January
2024.
On Monday 22 January we had 4 HDD all on different server with I/O-error
because of some damage sectors, the OSD is hybrid so the DB is on SSD, 5
HDD share 1 SSD.
I set the OSD out, ceph osd out 223 269 290 318
eed to your local registry and specify it,
some details are here
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-environment
The containers themself don't need to download anything at start.
--
Kai Stian Olstad
___
ceph-users
t ceph.conf and
the admin key.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
incy/rados/operations/monitoring-osd-pg/#monitoring-pg-states
https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/
https://docs.ceph.com/en/latest/dev/placement-group/#user-visible-pg-states
--
Kai Stian Olstad
___
ceph-users mailing li
the whole host. I would very much
appreciate any suggestions.
If you look closely at 53906 you'll see it's a duplicate of
https://tracker.ceph.com/issues/53907
In there you have the fix and a workaround until next minor is released.
--
Kai St
sion by cephadmin install
What should I do?
The easiest is use tell, then you can run it on any node that have access to
ceph.
ceph tell osd.8 dump_historic_ops
ceph tell osd.8 help
will give you all you can do with tell.
--
Kai Stian Olstad
__
documentation, have you tried that?
https://docs.ceph.com/en/reef/radosgw/config-ref/#lifecycle-settings
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
-10-09T07:59:01.303+
This log say that it's bash with PID 4019 that is creating the log
entry.
Maybe start there, check what what other thing you are running on the
server that creates this messages.
--
Kai Stian Olstad
___
ceph-users mailing
a nice script that do the
steps Wes explained above
https://github.com/cernceph/ceph-scripts/blob/master/tools/scrubbing/autorepair.sh
that you might want to take a look at.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscr
e of thumb, but with so large disk you could or maybe should
consider having a higher number of PGs per OSD.
[1]
https://docs.ceph.com/en/quincy/rados/operations/placement-groups/#viewing-pg-scaling-recommendations
--
Kai Stian Olstad
___
ceph-users ma
domain set to
host, it's my understanding no data will be migrated/backfilled when a node
fails.
The reason is that there is nowhere to copy the data to, to fulfill the crush
rule
one copy on 3 different hosts.
--
Kai Stian Olstad
___
ceph-users ma
to do this
ceph config-key get mgr/cephadm/spex.rgw.pech | jq -r
.spec.spec.rgw_frontend_ssl_certificate | ceph config-key set rgw/cert/rgw.pech
-
ceph orch restart rgw.pech
My claim is that Ceph should update "rgw/cert/rgw.pech" when
"mgr/cephadm/spex.rgw.pech" is
d here they set rgw_max_concurrent_requests to 10240
https://www.seagate.com/content/dam/seagate/migrated-assets/www-content/solutions/partners/red-hat/_shared/files/st-seagate-rhcs5-detail-f29951wg-202110-en.pdf
So I think the only way to find out it to increase it and see what happens.
--
K
PLACEMENT
rgw.pech ?:443 7/7 4m ago 22h label:cog
But running
ceph orch redeploy rgw.pech
will redeploy all 7 RGW, and would be the same as
ceph orch daemon redeploy rgw.pech.pech-mon-3.upnvrd
but only redeploy one of them.
From: Kai Stian Olstad
The certificate is about to
eased, but could be increase it to
2048.
But the recommended action was to add more gateways instead.
[1]
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_max_concurrent_requests
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-use
ones about to expire.
I have looked in the documentation and can't find how to update the certificate
for RGW.
Can anyone shed some light on how to replace the certificate?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.
On Wed, Mar 01, 2023 at 08:39:56AM -0500, Daniel Gryniewicz wrote:
We're actually writing this for RGW right now. It'll be a bit before
it's productized, but it's in the works.
Just curious, what is the use cases for this feature?
S3 against CephFS?
Hi
Just a follow up, the issue was solved by running command
ceph pg 404.1ff mark_unfound_lost delete
-
Kai Stian Olstad
On 04.01.2023 13:00, Kai Stian Olstad wrote:
Hi
We are running Ceph 16.2.6 deployed with Cephadm.
Around Christmas OSD 245 and 327 had about 20 read error so I set
";
Does anyone have any idea on how to resolv the problem?
Any help is much appreciated.
-
Kai Stian Olstad
::
ceph-s.txt
::
ceph -s
---
cluster:
id: d13c6b81-51ee-4d22-84e9-456f9307296c
health: HEALTH_ERR
1/416287125 ob
olumens_'
I have found this by looking at how Openstack does it and some trial and
error.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 16.11.2022 13:21, E Taka wrote:
gmail marks too many messages on this mailing list as spam.
You can fix that by creating a filter in Gmail for ceph-users@ceph.io
and check the "Never send it to Spam".
--
Kai Stian Olstad
___
ceph-use
On 16.11.2022 00:25, Daniel Brunner wrote:
are my mails not getting through?
is anyone receiving my emails?
You can check this yourself by checking the archives
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/
If you see your mail there, they are getting through.
--
Kai Stian
x27;t support SMART or any status on SAS disk today, I only get
the message "No SMART data available".
I have gathered "smartctl -x --json=vo" log for the 6 types of SAS this
I have in my possession.
You can find them here if interested [1]
[1] https://gitl
And is the keyring file readable for the user running the ceph commands?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
.")[0] }}
This works because the first part of the instance name is the hostname.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
quot; and check status every 2 seconds for 2 minutes, no
degraded messages.
The only new messages in ceph -s was 12 remapped pgs and "11
active-remapped+backfilling" and "1 active+remapped+backfill_wait"
Previously I had to set all osd(15 disks) on a host to out and there was
n
pool, but the degradation problems
also
go away when we turn off the balancer.
So this goes a long way of confirming there are something wrong with the
balancer since we now see it on two different installation.
--
Kai Stian Olstad
___
ceph-
on the osd too and it will set the correct
permissions.
# ceph auth get-or-create client.user2 mon 'profile rbd' osd 'profile rbd
pool=rbd namespace=user2' -o /etc/ceph/client.user2.keyring
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
request to me :)
Of course, I would never refer someone to send a feature request to a
person even if you were a Ceph developer, I would consider that rude,
the tracker exist for that :-)
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users
be, It's according to spec but you
can always put in a feature request.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
anyone know it's correct to return 416 on 0 byte object with range
or should this be considered a bug in Ceph.
[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-
On 15.03.2022 10:10, Jimmy Spets wrote:
Thanks for your reply.
I have two things that I am unsure of:
- Is the OSD UUID the same for all OSD:s or should it be unique for
each?
It's unique and generated when you run ceph-volume lvm prepare or add an
OSD.
You can find OSD UUID/FSID for existi
On 11.03.2022 14:04, Ilya Dryomov wrote:
On Fri, Mar 11, 2022 at 8:04 AM Kai Stian Olstad
wrote:
Isn't namespace supported with erasure encoded pools?
RBD images can't be created in EC pools, so attempting to create RBD
namespaces there is pointless. The way to store RBD image
naming.
root@osd-host:~# cephadm deploy --fsid --name osd.152
--config-json config-osd.152.json --osd-fsid
517213f3-0715-4d23-8103-6a34b1f8ef08
And then the OSD should be back up and running.
This is the way I have found to do OSD replacement, it might be an
easier way of doing it but I ha
ailed to add namespace: (95) Operation not supported
Isn't namespace supported with erasure encoded pools?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
with
> no user data at all.
When you have DB on a separate disk the DB size count towards total size of the
osd. But this DB space is considered used so you will see a lot of used space.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@c
change
I can't find the feed URL.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
.com/en/pacific/install/index.html
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
is issue
https://tracker.ceph.com/issues/51667
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
n an isolated network.
[1] https://github.com/ceph/ceph/blob/v16.2.7/src/cephadm/cephadm#L50-L61
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
empty except for one pool with one
PG created by Cephadm.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
(Xeon 5222 3.8 GHz) and non of the cores was
saturated,
and network statistics show no errors or drops.
So perhaps there is a bug in the balancer code?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 16.09.2021 15:51, Josh Baergen wrote:
I assume it's the balancer module. If you write lots of data quickly
into the cluster the distribution can vary and the balancer will try
to even out the placement.
The balancer won't cause degradation, only misplaced objects.
Since I'm trying to test
Degraded data redundancy: 4315/7005845 objects degraded (0.062%), 2
pgs degraded, 4 pgs undersized
So my question is, it normal that Ceph report degraded under normal use?
or do I have a problem somewhere that I need to investigate?
--
Kai Stian Olstad
___
oard/issues/new ?
Found a issue reported under MGR
https://tracker.ceph.com/issues/52028 - mgr/dashboard: Incorrect MTU
mismatch warning
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
difference but haven’t gotten around to
it.
If you look at the message it says which interface it is.
It does check and report on all the interfaces, even those that is in
DOWN state which it shouldn't.
--
Kai Stian Olstad
___
ceph-users mailing
Hi Peter
Please remember to include the list address in your reply.
I will not trim so people on the list can read you answer.
On 29.07.2021 12:43, Peter Childs wrote:
On Thu, 29 Jul 2021 at 10:37, Kai Stian Olstad
wrote:
A little disclaimer, I have never used multipath with Ceph.
On
sure that the LVM only scan the multipath device
paths and not the individual disk the OS sees.
[1] https://docs.ceph.com/en/latest/cephadm/osd/#creating-new-osds
[2]
https://docs.ceph.com/en/latest/cephadm/osd/#advanced-osd-service-specifications
--
Kai Stian Olstad
On 22.07.2021 13:56, Kai Stian Olstad wrote:
Hi
I have a warning that says
"1 stray daemon(s) not managed by cephadm"
What i did is the following.
I have 3 nodes that the mon should run on, but because of a bug in
16.2.4 I couldn't run on then since they are in different subne
;^"' | egrep -v "osd|node|crash" | sort
"alertmanager.pech-mds-1"
"grafana.pech-cog-2"
"mgr.pech-mon-1.ptrsea"
"mgr.pech-mon-2.mfdanx"
"mon.pech-mon-1"
"mon.pech-mon-2"
"mon.pech-mon-3"
"prometheus.pech-md
a breaking install/upgrade issue like this
has been allowed to sit)
You and me both.
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
LSize
osd-block-9227e8ae-92eb-429e-9c7f-d4a2b75afb8e
ceph-46f42262-d3dc-4dc3-8952-eec3e4a2c178 -wi-ao 12.47t
osd-block-2da790bc-a74c-41da-8772-3b8aac77001c
ceph-block-1b5ad7e7-2e24-4315-8a05-7439ab782b45 -wi-ao 12.47t
The fist one is the new OSD and the seco
redeployed on one node. This spec file results in three
standalone OSDs! Without the other filters this won't work as
expected, it seems. I'll try again on Pacific with the same test and
see where that goes.
This spec did worked for me when I initially deployed with Octopus
15.2.5.
But it doesn't find any disk.
I also tried this, but with the same result.
service_type: osd
service_id: hdd
placement:
host_pattern: 'pech-hd-*'
block_db_size: 360G
data_devices:
rotational: 1
db_devices:
rotational: 0
filter_logic: AND
I'll try again with Octopus to see
r: Creating volume group
backup
"/etc/lvm/backup/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b"
(seqno 62).
INFO:cephadm:/usr/bin/podman:stderr --> Zapping successful for: /dev/ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b/osd-block-db-449bd001-eb32-46de-ab80-a1cbcd293d6
fficient space (<10 extents) on vgs, LVM detected, locked
pech-hd-7 /dev/sds hdd SEAGATE ST14000NM016813.7T No
Insufficient space (<10 extents) on vgs, LVM detected, locked
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
|DATA |DB |WAL |
+-+--+-+--++-+
|osd |hdd |pech-hd-7|/dev/sdt |- |-|
+-+--+-+--++-+
To me I looks like Cephadm can't use/find the free space on the VG and
use that as a new LV for the OSD.
--
Kai Stian Olstad
___
tried overwriting the LV with
/dev/zero, but that did solve it.
So I guess with wipe do you mean delete the LV with lvremove?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
I guess this is the wrong way to do it, but I can't find the answer in
the documentation.
So how can I replace this failed disk in Cephadm?
--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
On 11.03.2021 15:47, Sebastian Wagner wrote:
yes
Am 11.03.21 um 15:46 schrieb Kai Stian Olstad:
To resolve it, could I just remove it with "cephadm rm-daemon"?
That worked like a charm, and the upgrade is resumed.
Thank you Sebastian.
--
Kai St
: "stopped",
"container_id": null,
"container_image_name":
"goharbor.example.com/library/ceph/ceph:v15.2.5",
"container_image_id": null,
"version": null,
"started": null,
"created": "2021-01-20
as above.
Looking at the source code it looks like it get daemons to upgrade from
mgr cache, so I restarted both mgr but still it tries to upgrade
osd.355.
Does anyone know how I can get the upgrade to continue?
--
Kai Stian Olstad
___
ceph-users m
88 matches
Mail list logo
