[ceph-users] Re: [cephadm] Questions regarding cephadm infra-as-code philosophy, containerization, and trixie availability

2025-05-15 Thread Fox, Kevin M
my 2 cents, for the enterprise, there is nothing better then being able to have codified all the best practices around the proper orchestration of something as complicated as a ceph cluster into an orchestration layer and done so in a way that can help prevent mistakes. Without the orchestrator

[ceph-users] Re: Heads up -- 18.2.6 regression

2025-05-01 Thread Fox, Kevin M
One of the referenced issues seems to indicate it may be in quincy too? does anyone know if thats true, and when it was introduced? Does it go back even farther? Thanks, Kevin From: Devender Singh Sent: Thursday, May 1, 2025 11:40 AM To: Alex Cc: Dan va

[ceph-users] Re: Understanding filesystem size

2025-01-14 Thread Fox, Kevin M
bulk is about tuning the pool assuming it will be closeish to full, not based on what utilization currently is. That way, as you add a bunch of data, it isn't constantly adding more pg's to the pool. From: Anthony D'Atri Sent: Tuesday, January 14, 2025 9

[ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time

2024-10-25 Thread Fox, Kevin M
That is super useful. Thank you so much for sharing! :) Kevin From: Frank Schilder Sent: Friday, October 25, 2024 8:03 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: pgs not deep-scrubbed in time and pgs not scrubbed in time Check twice before you

[ceph-users] Re: Issue Replacing OSD with cephadm: Partition Path Not Accepted

2024-09-03 Thread Fox, Kevin M
I thought bluestore stored that stuff in non lvm mode? From: Robert Sander Sent: Monday, September 2, 2024 11:35 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: Issue Replacing OSD with cephadm: Partition Path Not Accepted Check twice before you clic

[ceph-users] Re: What's the best way to add numerous OSDs?

2024-08-06 Thread Fox, Kevin M
some kernels (el7?) lie about being jewel until after they are blocked from connecting at jewel. then they report newer. Just fyi. From: Anthony D'Atri Sent: Tuesday, August 6, 2024 5:08 PM To: Fabien Sirjean Cc: ceph-users Subject: [ceph-users] Re: What'

[ceph-users] Re: Bug Found in Reef Releases - Action Required for pg-upmap-primary Interface Users

2024-05-29 Thread Fox, Kevin M
How do you know if its safe to set `require-min-compat-client=reef` if you have kernel clients? Thanks, Kevin From: Laura Flores Sent: Wednesday, May 29, 2024 8:12 AM To: ceph-users; dev; clt Cc: Radoslaw Zarzynski; Yuri Weinstein Subject: [ceph-users] B

[ceph-users] Re: Ceph Pacific bluefs enospc bug with newly created OSDs

2023-06-21 Thread Fox, Kevin M
Does quincy automatically switch existing things to 4k or do you need to do a new ost to get the 4k size? Thanks, Kevin From: Igor Fedotov Sent: Wednesday, June 21, 2023 5:56 AM To: Carsten Grommel; ceph-users@ceph.io Subject: [ceph-users] Re: Ceph Pacif

[ceph-users] Re: BlueStore fragmentation woes

2023-05-30 Thread Fox, Kevin M
-30T18:35:22.826+ 7fe190013700 0 bluestore(/var/lib/ceph/osd/ceph-183) probe -20: 0, 0, 0 Thanks, Kevin From: Fox, Kevin M Sent: Thursday, May 25, 2023 9:36 AM To: Igor Fedotov; Hector Martin; ceph-users@ceph.io Subject: Re: [ceph-users] Re: BlueStore

[ceph-users] Re: BlueStore fragmentation woes

2023-05-25 Thread Fox, Kevin M
720 if that is the right query, then I'll gather the metrics, restart and gather some more after and let you know. Thanks, Kevin From: Igor Fedotov Sent: Thursday, May 25, 2023 9:29 AM To: Fox, Kevin M; Hector Martin; ceph-users@ceph.io Subject:

[ceph-users] Re: BlueStore fragmentation woes

2023-05-25 Thread Fox, Kevin M
If you can give me instructions on what you want me to gather before the restart and after restart I can do it. I have some running away right now. Thanks, Kevin From: Igor Fedotov Sent: Thursday, May 25, 2023 9:17 AM To: Fox, Kevin M; Hector Martin

[ceph-users] Re: BlueStore fragmentation woes

2023-05-25 Thread Fox, Kevin M
Is this related to https://tracker.ceph.com/issues/58022 ? We still see run away osds at times, somewhat randomly, that causes runaway fragmentation issues. Thanks, Kevin From: Igor Fedotov Sent: Thursday, May 25, 2023 8:29 AM To: Hector Martin; ceph-us

[ceph-users] Re: Ceph Failure and OSD Node Stuck Incident

2023-03-30 Thread Fox, Kevin M
I've seen this twice in production on two separate occasions as well. one osd gets stuck. a bunch of pg's go into laggy state. ceph pg dump | grep laggy shows all the laggy pg's share the same osd. Restarting the affected osd restored full service. From

[ceph-users] Re: s3 compatible interface

2023-03-21 Thread Fox, Kevin M
Will either the file store or the posix/gpfs filter support the underlying files changing underneath so you can access the files either through s3 or by other out of band means (smb, nfs, etc)? Thanks, Kevin From: Matt Benjamin Sent: Monday, March 20, 2

[ceph-users] Re: s3 compatible interface

2023-03-06 Thread Fox, Kevin M
+1. If I know radosgw on top of cephfs is a thing, I may change some plans. Is that the planned route? Thanks, Kevin From: Daniel Gryniewicz Sent: Monday, March 6, 2023 6:21 AM To: Kai Stian Olstad Cc: ceph-users@ceph.io Subject: [ceph-users] Re: s3 comp

[ceph-users] Re: s3 compatible interface

2023-02-28 Thread Fox, Kevin M
Minio no longer lets you read / write from the posix side. Only through minio itself. :( Haven't found a replacement yet. If you do, please let me know. Thanks, Kevin From: Robert Sander Sent: Tuesday, February 28, 2023 9:37 AM To: ceph-users@ceph.io Su

[ceph-users] Re: Ceph Upgrade path

2023-02-01 Thread Fox, Kevin M
We successfully did ceph-deploy+octopus+centos7 -> (ceph-deploy unsupported)+octopus+centos8stream (using leap) -> (ceph-deploy unsupported)+pacific+centos8stream -> cephadm+pacific+centos8stream Everything in place. Leap was tested repeatedly till the procedure/sideeffects were very well know

[ceph-users] Re: BlueFS spillover warning gone after upgrade to Quincy

2023-01-12 Thread Fox, Kevin M
If you have prometheus enabled, the metrics should be in there I think? Thanks, Kevin From: Peter van Heusden Sent: Thursday, January 12, 2023 6:12 AM To: ceph-users@ceph.io Subject: [ceph-users] BlueFS spillover warning gone after upgrade to Quincy Chec

[ceph-users] Re: 2 pgs backfill_toofull but plenty of space

2023-01-10 Thread Fox, Kevin M
What else is going on? (ceph -s). If there is a lot of data being shuffled around, it may just be because its waiting for some other actions to complete first. Thanks, Kevin From: Torkil Svensgaard Sent: Tuesday, January 10, 2023 2:36 AM To: ceph-users@

[ceph-users] Remove radosgw entirely

2022-12-13 Thread Fox, Kevin M
Is there any problem removing the radosgw and all backing pools from a cephadm managed cluster? Ceph won't become unhappy about it? We have one cluster with a really old, historical radosgw we think would be better to remove and someday later, recreate fresh. Thanks, Kevin _

[ceph-users] Re: Ceph upgrade advice - Luminous to Pacific with OS upgrade

2022-12-06 Thread Fox, Kevin M
We went on a couple clusters from ceph-deploy+centos7+nautilus to cephadm+rocky8+pacific using ELevate as one of the steps. Went through octopus as well. ELevate wasn't perfect for us either, but was able to get the job done. Had to test it carefully on the test clusters multiple times to get th

[ceph-users] Re: Tuning CephFS on NVME for HPC / IO500

2022-12-01 Thread Fox, Kevin M
if its this: http://www.acmemicro.com/Product/17848/Kioxia-KCD6XLUL15T3---15-36TB-SSD-NVMe-2-5-inch-15mm-CD6-R-Series-SIE-PCIe-4-0-5500-MB-sec-Read-BiCS-FLASH-TLC-1-DWPD its listed as 1 DWPD with a 5 year warranty. So should be ok. Thanks, Kevin From: Rob

[ceph-users] Re: osd set-require-min-compat-client

2022-11-30 Thread Fox, Kevin M
When we switched (Was using the compat balancer previously), I: 1. turned off the balancer 2. forced the client minimum (new centos7 clients are ok being forced to luminious even though they report as jewel. There's an email thread elsewhere describing it) 3. slowly reweighted the crush compat w

[ceph-users] Re: Configuring rgw connection timeouts

2022-11-17 Thread Fox, Kevin M
I think you can do it like: ``` service_type: rgw service_id: main service_name: rgw.main placement: label: rgwmain spec: config: rgw_keystone_admin_user: swift ``` ? From: Thilo-Alexander Ginkel Sent: Thursday, November 17, 2022 10:21 AM To: Case

[ceph-users] Re: How to monitor growing of db/wal partitions ?

2022-11-14 Thread Fox, Kevin M
There should be prom metrics for each. Thanks, Kevin From: Christophe BAILLON Sent: Monday, November 14, 2022 10:08 AM To: ceph-users Subject: [ceph-users] How to monitor growing of db/wal partitions ? Check twice before you click! This email originated

[ceph-users] Re: Is it a bug that OSD crashed when it's full?

2022-11-01 Thread Fox, Kevin M
If its the same issue, I'd check the fragmentation score on the entire cluster asap. You may have other osds close to the limit and its harder to fix when all your osds cross the line at once. If you drain this one, it may push the other ones into the red zone if your too close, making the probl

[ceph-users] Re: cephadm trouble with OSD db- and wal-device placement (quincy)

2022-11-01 Thread Fox, Kevin M
I haven't done it, but had to read through the documentation a couple months ago and what I gathered was: 1. if you have a db device specified but no wal device, it will put the wal on the same volume as the db. 2. the recommendation seems to be to not have a separate volume for db and wal if on

[ceph-users] Re: monitoring drives

2022-10-14 Thread Fox, Kevin M
Would it cause problems to mix the smartctl exporter along with ceph's built in monitoring stuff? Thanks, Kevin From: Wyll Ingersoll Sent: Friday, October 14, 2022 10:48 AM To: Konstantin Shalygin; John Petrini Cc: Marc; Paul Mezzanini; ceph-users Subjec

[ceph-users] Re: v15-2-14-octopus no docker images on docker hub ceph/ceph ?

2021-08-20 Thread Fox, Kevin M
We launch a local registry for cases like these and mirror the relevant containers there. This keeps copies of the images closer to the target cluster and reduces load on the public registries. Its not that much different from mirroring a yum/apt repo locally to speed up access. For large cluste

[ceph-users] Re: Ceph with BGP?

2021-07-06 Thread Fox, Kevin M
I'm not aware of any directly, but I know rook-ceph is used on Kubernetes, and Kubernetes is sometimes deployed with BGP based SDN layers. So there may be a few deployments that do it that way. From: Martin Verges Sent: Monday, July 5, 2021 11:23 PM To:

[ceph-users] Re: Ceph connect to openstack

2021-06-30 Thread Fox, Kevin M
https://docs.ceph.com/en/latest/rbd/rbd-openstack/ From: Szabo, Istvan (Agoda) Sent: Wednesday, June 30, 2021 9:50 AM To: Ceph Users Subject: [ceph-users] Ceph connect to openstack Check twice before you click! This email originated from outside PNNL. H

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-25 Thread Fox, Kevin M
Orchestration is hard, especially with every permutation. The devs have implemented what they feel is the right solution for their own needs from the sound of it. The orchestration was made modular to support non containerized deployment. It just takes someone to step up and implement the permut

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-24 Thread Fox, Kevin M
I bumped into this recently: https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/ :) Kevin From: Sage Weil Sent: Thursday, June 24, 2021 2:06 PM To: Stefan Kooman Cc: Nico Schottelius; Kai Börnert; Marc; ceph-users Subject: [ce

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-24 Thread Fox, Kevin M
I've actually had rook-ceph not proceed with something that I would have continued on with. Turns out I was wrong and it was right. Its checking was more through then mine. Thought that was pretty cool. It eventually cleared itself and finished up. For a large ceph cluster, the orchestration is

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-02 Thread Fox, Kevin M
Ultimately, that's what a container image is. From the outside, its a statically linked binary. From the inside, it can be assembled using modular techniques. The best thing about it, is you can use container scanners and other techniques to gain a lot of the benefits of that modularity still. P

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-02 Thread Fox, Kevin M
While there are many reasons containerization helps, I'll just touch on one real quick that is relevant to the conversation. Orchestration. Implementing orchestration of an entire clustered software with many different: * package managers * dependency chains * init systems * distro specific quir

[ceph-users] Re: Why you might want packages not containers for Ceph deployments

2021-06-02 Thread Fox, Kevin M
Debating containers vs packages is like debating systemd vs initrd. There are lots of reasons why containers (and container orchestration) are good for deploying things, including ceph. Repeating them in each project every time it comes up is not really productive. I'd recommend looking at why c

[ceph-users] Re: cephfs vs rbd vs rgw

2021-05-25 Thread Fox, Kevin M
The quick answer, is they are optimized for different use cases. Things like relational databases (mysql, postgresql) benefit from the performance that a dedicated filesystem can provide (rbd). Shared filesystems are usually counter indicated with such software. Shared filesystems like cephfs a

[ceph-users] Re: ceph-ansible in Pacific and beyond?

2021-03-17 Thread Fox, Kevin M
There are a lot of benefits to containerization that is hard to do without it. Finer grained ability to allocate resources to services. (This process gets 2g of ram and 1 cpu) Security is better where only minimal software is available within the container so on service compromise its harder to e

[ceph-users] Re: ceph osd df results

2021-02-11 Thread Fox, Kevin M
+1 From: Marc Sent: Thursday, February 11, 2021 12:09 PM To: ceph-users Subject: [ceph-users] ceph osd df results Check twice before you click! This email originated from outside PNNL. Should the ceph osd df results not have this result for every device

[ceph-users] Re: radosgw bucket index issue

2021-02-02 Thread Fox, Kevin M
Ping From: Fox, Kevin M Sent: Tuesday, December 29, 2020 3:17 PM To: ceph-users@ceph.io Subject: [ceph-users] radosgw bucket index issue We have a fairly old cluster that has over time been upgraded to nautilus. We were digging through some things and

[ceph-users] radosgw bucket index issue

2020-12-29 Thread Fox, Kevin M
We have a fairly old cluster that has over time been upgraded to nautilus. We were digging through some things and found 3 bucket indexes without a corresponding bucket. They should have been deleted but somehow were left behind. When we try and delete the bucket index, it will not allow it as t