[ceph-users] Re: v19.2.3 Squid released

___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v19.2.3 Squid released

___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Heads up: bad drive for Ceph Western Digital Ultrastar DC HC560

Hi, > On 4 Jul 2025, at 13:14, Marc wrote: > > How is it worse than any other hdd of that size? At the moment we have just under 3000pcs of Toshiba MG10ACA and we have not registered any such issues k ___ ceph-users mailing list -- ceph-users@ceph

[ceph-users] Heads up: bad drive for Ceph Western Digital Ultrastar DC HC560

Hi, When deploying new project, we discovered poor performance [1] of hard drives Western Digital Ultrastar DC HC560 (20TB, WUH722020BLE6L4). It consists in the fact that with an object size of less than 128k, the speed of the Ceph pool is equivalent to a USB flash drive. Be vigilant when make

[ceph-users] Re: Newly added node osds getting full before rebalance finished

Hi, > On 27 May 2025, at 10:54, Szabo, Istvan (Agoda) > wrote: > > Some status update, it's finished with 3x times stop and start the rebalance. > Would be interesting to know what is the extra data generated on the new osds > during remapped pg allocation at rebalance. I stopped when the osd

[ceph-users] Re: ceph reef on AlmaLinux9 (and the dependency problem on openssl-libs)

Hi, Thanks for the perfect overview of the Reef release! I'll steal this as is for the slide, for another overview of why it's important to have an update strategy. Sometimes folks don't understand why our 75 clusters are using the Nautilus or Pacific release Thanks, k Sent from my iPhone >

[ceph-users] Re: rpm-quincy

Hi, > On 20 May 2025, at 11:26, farhad kh wrote: > > Hi, I need to install ceph-common from the Quincy repository, but I'm > getting this error: > --- > Ceph x86_64 > 0.0 B/s | 0 B 00:01 > Errors during downloading metadata for repository 'Ceph': > - Status code: 404

[ceph-users] Re: v18.2.7 Reef released

Hi, > On 8 May 2025, at 23:12, Erwin Bogaard wrote: > > It looks like there is an issue with the package-based 18.2.7 release, when > upgrading from 18.2.6 on el9. > There seems to be a new (unfulfilled) dependency that prevents the packages > from installing: > > Problem 1: cannot install the

[ceph-users] Re: Question about compatibility of librdb and backend cluster

Hi, > On 21 Apr 2025, at 15:02, 段世博 wrote: > > Hi everyone, I would like to ask about the compatibility of librdb > versions. If the client version (librbd) used is 17.2.8, but the backend > cluster is 16.2.15 or 15.2.17, or even lower versions, can it be accessed > normally? Any reply will be v

[ceph-users] Re: nodes with high density of OSDs

Hi, > On 11 Apr 2025, at 10:53, Alex from North wrote: > > Hello Tim! First of all, thanks for the detailed answer! > Yes, probably in set up of 4 nodes by 116 OSD it looks a bit overloaded, but > what if I have 10 nodes? Yes, nodes itself are still heavy but in a row it > seems to be not that

[ceph-users] Re: FS not mount after update to quincy

Hi, > On 11 Apr 2025, at 09:59, Iban Cabrillo wrote: > > 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300:/ /cephvmsfs ceph > name=cephvmsfs,secretfile=/etc/ceph/cephvmsfs.secret,noatime,mds_namespace=cephvmsfs,_netdev > 0 0 Try add the ms_mode option, because you use msgr2 protocol. For example,

[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback

Hi, You can always consult with Releases page [1] Thanks, k [1] https://github.com/prometheus-community/smartctl_exporter/releases Sent from my iPhone > On 9 Apr 2025, at 17:51, Anthony D'Atri wrote: > > Unless something has changed with smartctl_exporter, there wasn’t working > support for

[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback

Hi, It's will be very nice, if this module will be removed. Everything that Ceph operator need can be covered via smartctl_exporter [1] Thanks, k [1] https://github.com/prometheus-community/smartctl_exporter Sent from my iPhone > On 8 Apr 2025, at 02:20, Yaarit Hatuka wrote: > > We would l

[ceph-users] Re: Ceph Tentacle release timeline — when?

Hi, The respondents are not confused in the topic, only once again it is highlighted that releases once every 9 months are not really necessary, considering that the team presented the last bugfix release in July of last year (now it is February). Regarding containers, once again the community

[ceph-users] Re: Ceph Tentacle release timeline — when?

I concur strongly with Matthew’s assessment. k Sent from my iPhone > On 6 Feb 2025, at 16:13, Matthew Leonard (BLOOMBERG/ 120 PARK) > wrote: > > Bloomberg is mainly agnostic to the time delay, obviously getting back in > alignment with OS releases is ideal. > > We cannot overstate our agree

[ceph-users] Re: RGW Exporter for Storage Class Metrics

Hi, > On 31 Jan 2025, at 17:25, Preisler, Patrick wrote: > > we would like to have a detailed usage report for our S3 Buckets. I already > installed this rgw exporter > https://github.com/blemmenes/radosgw_usage_exporter and it does provide > useful information about the buckets and the amoun

[ceph-users] Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard

___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v17.2.8 Quincy released - failed on Debian 11

Hi, > On 26 Nov 2024, at 16:10, Matthew Darwin wrote: > > I guess there is a missing dependency (which really should be > auto-installed), which is not also documented in the release notes as a new > requirement. This seems to fix it: This caused by [1], the fix was not backported to quincy,

[ceph-users] Re: All monitors fall down simultaneously when I try to map rbd on client

Hi, > On 27 Sep 2024, at 14:59, Alex from North wrote: > > By increasing debulg level I found out the following but have no idea how to > fix this issue. > > ``` > src/osd/OSDMap.cc: 3242: FAILED ceph_assert(pg_upmap_primaries.empty()) > ``` > > There is only one topic in google and with no a

[ceph-users] Re: Mds daemon damaged - assert failed

Hi, The [2] is the fix for [1] and should be backported? Currently fields are not filled, so no one knows that backports are needed k > On 27 Sep 2024, at 11:01, Frédéric Nass > wrote: > > Hi George, > > Looks like you hit this one [1]. Can't find the fix [2] in Reef release notes > [3].

[ceph-users] Re: High usage (DATA column) on dedicated for OMAP only OSDs

Hi, > On 19 Sep 2024, at 18:26, Joshua Baergen wrote: > > Whenever we've seen osdmaps not being trimmed, we've made sure that > any down OSDs are out+destroyed, and then have rolled a restart > through the mons. As of recent Pacific at least this seems to have > reliably gotten us out of this si

[ceph-users] Re: High usage (DATA column) on dedicated for OMAP only OSDs

Hi, > On 19 Sep 2024, at 12:33, Igor Fedotov wrote: > > osd_target_transaction_size should control that. > > I've heard of it being raized to 150 with no obvious issues. Going beyond is > at your own risk. So I'd suggest to apply incremental increase if needed. Thanks! Now much better k __

[ceph-users] Re: High usage (DATA column) on dedicated for OMAP only OSDs

Hi Igor, > On 18 Sep 2024, at 18:22, Igor Fedotov wrote: > > I recall a couple of cases when permanent osdmap epoch growth has been > filling OSD with relevant osd map info. Which could be tricky to catch. > > Please run 'ceph tell osd.N status" for a couple of affected OSDs twice > within e.

[ceph-users] Re: ceph-mgr perf throttle-msgr - what is caused fails?

Hi, Increasing this value to 30 is the only thing I could do at the moment k Sent from my iPhone > On 13 Sep 2024, at 16:49, Eugen Block wrote: > > I remember having a prometheus issue quite some time ago, it couldn't handle > 30 nodes or something, not really a big cluster. But we needed to

[ceph-users] Re: ceph-mgr perf throttle-msgr - what is caused fails?

As I said before, currently Prometheus module performance degradation is only one _visible_ issues. I named things like is as indicator (of feature problem's) k Sent from my iPhone > On 12 Sep 2024, at 23:18, Eugen Block wrote: > > But did you notice any actual issues or did you just see tha

[ceph-users] Re: ceph-mgr perf throttle-msgr - what is caused fails?

gt;> "max": 104857600, > > So it probably doesn't have any visible impact, does it? But the values are > not that far apart, maybe they burst sometime, leading to the fail_fail > counter to increase? Do you have that monitored? > > Thanks, > Eugen > &g

[ceph-users] ceph-mgr perf throttle-msgr - what is caused fails?

Hi, seems something in mgr is throttle due val > max. I'm right? root@mon1# ceph daemon /var/run/ceph/ceph-mgr.mon1.asok perf dump | jq '."throttle-msgr_dispatch_throttler-mgr-0x55930f4aed20"' { "val": 104856554, "max": 104857600, "get_started": 0, "get": 9700833, "get_sum": 6544522184

[ceph-users] Re: Connecting A Client To 2 Different Ceph Clusters

Hi, I don't think kernel read any files. Maybe only with some userland helper. In any case, it's up to you, but for the kernel ceph client to work, nothing but the kernel is needed, bindings in the form of files and user space utilities only complicate the system, like for me k Sent from my i

[ceph-users] Re: Connecting A Client To 2 Different Ceph Clusters

Hi, > On 24 Aug 2024, at 10:57, duluxoz wrote: > > How do I set up the ceph.conf file(s) on my clients so that I can use > /etc/fstab to connect to each CephFS or each Ceph Cluster? You can try to setup your fstab like this: 10.0.0.1:3300,10.0.0.2:3300,10.0.0.3:3300:/folder1 /mnt ceph name=

[ceph-users] Re: Pull failed on cluster upgrade

Hi, > On 7 Aug 2024, at 10:31, Nicola Mori wrote: > > Unfortunately I'm on bare metal, with very old hardware so I cannot do much. > I'd try to build a Ceph image based on Rocky Linux 8 if I could get the > Dockerfile of the current image to start with, but I've not been able to find > it. Ca

[ceph-users] Re: ceph 18.2.4 on el8?

Hi, > On 30 Jul 2024, at 00:51, Christopher Durham > wrote: > > I see that 18.2.4 is out, in rpm for el9 at: > http://download.ceph.com/rpm-18.2.4/ Are there any plans for an '8' version? > One of my clusters is not yet ready to update to Rocky 9. We will update to 9

[ceph-users] Re: ceph 18.2.4 on el8?

Hi, > On 30 Jul 2024, at 00:51, Christopher Durham wrote: > > I see that 18.2.4 is out, in rpm for el9 at: > http://download.ceph.com/rpm-18.2.4/ Are there any plans for an '8' version? > One of my clusters is not yet ready to update to Rocky 9. We will update to 9 > moving forward but this i

[ceph-users] Re: Reef 18.2.4 EL8 packages ?

Hi, > On 26 Jul 2024, at 20:22, Josh Durgin wrote: > > We didn't want to stop building on Centos 8, but the way it went end of > life and stopped doing any security updates forced our hand. See this > thread for details [0]. > > Essentially this made even building and testing with Centos 8 infe

[ceph-users] Re: Reef 18.2.4 EL8 packages ?

Hi, > On 25 Jul 2024, at 14:39, Noe P. wrote: > > I find this 18.2.4 release highly confusing. > > Can anyone please confirm that EL8 packages are no more supported ? > The small remark in the release notes > > qa/distros: remove centos 8 from supported distros (pr#57932, Guillaume > Abrioux

[ceph-users] Re: [Ceph-announce] v18.2.4 Reef released

Hi, > On 25 Jul 2024, at 00:12, Yuri Weinstein wrote: > > We're happy to announce the 4th release in the Reef series. The repo of reef now returns 404 > GET /rpm-reef/el8/SRPMS/ HTTP/1.1 > Host: download.ceph.com > < HTTP/1.1 404 Not Found If change the repo to previous version, the answer 20

[ceph-users] Re: Unable to mount with 18.2.2

Hi, > On 17 Jul 2024, at 10:21, Frédéric Nass > wrote: > > Seems like msgr v2 activation did only occur after all 3 MONs were redeployed > and used RocksDB. Not sure why this happened though. For a work with msgr v2 only, you need to specify ms_mode to prefer-crc, at least. For example of fst

[ceph-users] Re: cephadm for Ubuntu 24.04

> On 11 Jul 2024, at 15:20, John Mulligan wrote: > > I'll ask to have backport PRs get generated. I'm personally pretty clueless > as > to how to process backports. The how-to described in this doc [1] > Thanks, I hadn't found that one. Added backport for squid release [2], as far as I unde

[ceph-users] Re: [RGW] Strange issue of multipart object

Xin chao, The Pacific latest (16.2.15) have multiple multipart issue fixes (for example [1]), I suggest to upgrade your release for start k [1] https://tracker.ceph.com/issues/56673 Sent from my iPhone > On 18 Jun 2024, at 10:32, Huy Nguyen wrote: > > Hi, > I'm using Ceph v16.2.13. Using `

[ceph-users] Re: Unable to Install librados2 18.2.0 on RHEL 7 from Ceph Repository

Hi, The last release for EL7 is Octopus (version 15), you try to catch version 18 k Sent from my iPhone > On 29 May 2024, at 22:34, abdel.doui...@gmail.com wrote: > > The Ceph repository at https://download.ceph.com/ does not seem to have the > librados2 package version 18.2.0 for RHEL 7. Th

[ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi Eugen > On 21 May 2024, at 15:26, Eugen Block wrote: > > step set_choose_tries 100 I think you should try to increase set_choose_tries to 200 Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 hosts, the value of 100 not worked for us k ___

[ceph-users] Re: Ceph client cluster compatibility

Hi, Yes, like it always do k Sent from my iPhone > On 2 May 2024, at 07:09, Nima AbolhassanBeigi > wrote: > > We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04. > Our ceph cluster version is 16.2.13 (pacific). > > The problem is that the ubuntu packages for the ceph

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

Hi, > On 19 Apr 2024, at 10:39, Pardhiv Karri wrote: > > Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 > number in lsof output and then after 2 disks deletion again it got an error > and checked lsof and which is above 35000. I'm not sure how to handle it. > I reboot

[ceph-users] Re: Ceph image delete error - NetHandler create_socket couldnt create socket

Hi, Your shell seems reached the default file discriptors limit (1024 mostly) and your cluster maybe more than 1000 OSD Try to set command `ulimit -n 10240` before rbd rm task k Sent from my iPhone > On 18 Apr 2024, at 23:50, Pardhiv Karri wrote: > > Hi, > > Trying to delete images in a C

[ceph-users] Re: Client kernel crashes on cephfs access

Hi Xiubo, Seems patch already landed to kernel 6.8.7, thanks! k Sent from my iPhone > On 18 Apr 2024, at 05:31, Xiubo Li wrote: > > Hi Konstantin, > > We have fixed it, please see > https://patchwork.kernel.org/project/ceph-devel/list/?series=842682&archive=both. > > - Xiubo __

[ceph-users] Re: Client kernel crashes on cephfs access

Hi, > On 9 Apr 2024, at 04:07, Xiubo Li wrote: > > Thanks for reporting this, I generated one patch to fix it. Will send it out > after testing is done. Trace from our users, but from mainline kernel. Look like as trace above kernel: [ cut here ] kernel: list_add corr

[ceph-users] Re: Impact of large PG splits

> On 10 Apr 2024, at 01:00, Eugen Block wrote: > > I appreciate your message, it really sounds tough (9 months, really?!). But > thanks for the reassurance :-) Yes, the total "make this project great again" tooks 16 month, I think. This my work First problem after 1M objects in PG was a del

[ceph-users] Re: Impact of large PG splits

Hi Eugene! I have a case, where PG with millions of objects, like this ``` root@host# ./show_osd_pool_pg_usage.sh | less | head id used_mbytes used_objects omap_used_mbytes omap_used_keys -- --- -- 17.c91 1213.24827

[ceph-users] Re: Ceph object gateway metrics

Hi, You can use the [2] exporter to achieve usage stats per user and per bucket, including quotas usage k Sent from my iPhone > On 26 Mar 2024, at 01:38, Kushagr Gupta wrote: > > 2. https://github.com/blemmenes/radosgw_usage_exporter ___ ceph-users

[ceph-users] Re: Laptop Losing Connectivity To CephFS On Sleep/Hibernation

Hi, Yes, this is generic solution for end users mounts - samba gateway k Sent from my iPhone > On 23 Mar 2024, at 12:10, duluxoz wrote: > > Hi Alex, and thanks for getting back to me so quickly (I really appreciate > it), > > So from what you said it looks like we've got the wrong solution.

[ceph-users] Re: RGW - tracking new bucket creation and bucket usage

Hi, > On 15 Mar 2024, at 01:07, Ondřej Kukla wrote: > > Hello I’m looking for suggestions how to track bucket creation over s3 api > and bucket usage (num of objects and size) of all buckets in time. > > In our RGW setup, we have a custom client panel, where like 85% percent of > buckets are

[ceph-users] Re: ceph metrics units

Hi, > On 14 Mar 2024, at 19:29, Denis Polom wrote: > > so metric itself is miliseconds and after division on _count it's in seconds? > > This is two metrics for long running averages [1], the query that produces "seconds" unit looks like this (irate(ceph_osd_op_r_latency_sum[1m]) / irate(ce

[ceph-users] Re: ceph metrics units

Hi, > On 14 Mar 2024, at 16:44, Denis Polom wrote: > > do you know if there is some table of Ceph metrics and units that should be > used for them? > > I currently struggling with > > ceph_osd_op_r_latency_sum > > ceph_osd_op_w_latency_sum > > if they are in ms or seconds? > > Any idea ple

[ceph-users] Re: Telemetry endpoint down?

Hi Greg Seems is up now, last report uploaded successfully Thanks, k Sent from my iPhone > On 11 Mar 2024, at 18:57, Gregory Farnum wrote: > > We had a lab outage Thursday and it looks like this service wasn’t > restarted after that occurred. Fixed now and we’ll look at how to prevent > that

[ceph-users] Telemetry endpoint down?

Hi, seems telemetry endpoint is down for a some days? We have connection errors from multiple places 1:ERROR Mar 10 00:46:10.653 [564383]: opensock: Could not establish a connection to telemetry.ceph.com:443 2:ERROR Mar 10 01:48:20.061 [564383]: opensock: Could not establish a connecti

[ceph-users] Re: Running dedicated RGWs for async tasks

Hi, Yes. You need to turn off gc, lc threads in config for your current (client side) RGW's. Then setup your 'async tasks' RGW without client traffic. No special configuration needed, only if I wanna tune gc, lc settings k Sent from my iPhone > On 7 Mar 2024, at 13:09, Marc Singer wrote: >

[ceph-users] Re: ceph Quincy to Reef non cephadm upgrade

Hi, Yes, you upgrade ceph-common package, then restart your mons k Sent from my iPhone > On 6 Mar 2024, at 21:55, sarda.r...@gmail.com wrote: > > My question is - does this mean I need to upgrade all ceph packages (ceph, > ceph-common) and restart only monitor daemon first? __

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

Hi, Don't aware about what is SW, but if this software works with Prometheus metrics format - why not. Anyway the exporters are open source, you can modify the existing code for your environment k Sent from my iPhone > On 6 Mar 2024, at 07:58, Michael Worsham wrote: > > This looks interest

[ceph-users] Re: Monitoring Ceph Bucket and overall ceph cluster remaining space

Hi, For RGW usage statistics you can use radosgw_usage_exporter [1] k [1] https://github.com/blemmenes/radosgw_usage_exporter Sent from my iPhone > On 6 Mar 2024, at 00:21, Michael Worsham wrote: > Is there an easy way to poll the ceph cluster buckets in a way to see how > much space is re

[ceph-users] Re: Size return by df

Hi, Yes you can, this controlled by option client quota df = false k Sent from my iPhone > On Feb 22, 2024, at 11:17, Albert Shih wrote: > > Is they are any way to keep the first answer ? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubsc

[ceph-users] Re: pacific 16.2.15 QE validation status

> > On Feb 7, 2024, at 16:59, Zakhar Kirpichenko wrote: > > Indeed, it looks like it's been recently reopened. Thanks for this! Hi, It was merged yesterday Thanks for the right noise, k ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscr

[ceph-users] Re: Ceph 16.2.14: ceph-mgr getting oom-killed

Hi, The backport to pacific was rejected [1], you may switch to reef, when [2] merged and released [1] https://github.com/ceph/ceph/pull/55109 [2] https://github.com/ceph/ceph/pull/55110 k Sent from my iPhone > On Jan 25, 2024, at 04:12, changzhi tan <544463...@qq.com> wrote: > > Is there an

[ceph-users] Re: Ceph Nautilous 14.2.22 slow OSD memory leak?

Hi, > On Jan 12, 2024, at 12:01, Frédéric Nass > wrote: > > Hard to tell for sure since this bug hit different major versions of the > kernel, at least RHEL's from what I know. In what RH kernel release this issue was fixed? Thanks, k ___ ceph-us

[ceph-users] Re: MDS recovery with existing pools

Good to hear that, Eugen! CC'ed Zac for a your docs mention k > On Dec 11, 2023, at 23:28, Eugen Block wrote: > > Update: apparently, we did it! > We walked through the disaster recovery steps where one of the steps was to > reset the journal. I was under the impression that the specified com

[ceph-users] Re: osdmaptool target & deviation calculation

Hi, This deviation is very soft. If u wanna do real upmaps you should use deviation 1 k Sent from my iPhone > On Nov 27, 2023, at 21:39, Robert Hish wrote: > > The result is many many OSDs with a deviation well above the > upmap_max_deviation which is at default: 5 _

[ceph-users] Re: CLT Meeting minutes 2023-11-23

Hi, > On Nov 23, 2023, at 16:10, Nizamudeen A wrote: > > RCs for reef, quincy and pacific > for next week when there is more time to discuss Just little noise: pacific is ready? 16.2.15 should be last release (at least that was the last plan), but [1] still not merged. Why now ticket is clos

[ceph-users] Re: Bug fixes in 17.2.7

Hi, > On Nov 20, 2023, at 19:24, Tobias Kulschewski > wrote: > > do you have a rough estimate of when this will happen? > > Not at this year I think. For now precedence for a 18.2.1 and last release of Pacific But you can request shaman build, and clone repo for your local usage k ___

[ceph-users] Re: Bug fixes in 17.2.7

Hi Tobias, This was not meged to Quincy yet [1] k [1] https://tracker.ceph.com/issues/59730 Sent from my iPhone > On Nov 20, 2023, at 17:50, Tobias Kulschewski > wrote: > > Just wanted to ask, if the bug with the multipart upload [1] has been fixed > in 17.2.7? _

[ceph-users] Re: HDD cache

Hi Peter, > On Nov 8, 2023, at 20:32, Peter wrote: > > Anyone experienced this can advise? You can try: * check for current cache status smartctl -x /dev/sda | grep "Write cache" * turn off write cache smartctl -s wcache-sct,off,p /dev/sda * check again smartctl -x /dev/sda | grep "Write

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, You need revert your packages from Quincy to Pacific. `dnf downgrade ceph-mon` command should help with this k Sent from my iPhone > On Oct 11, 2023, at 03:22, Waywatcher wrote: > > I am unable to get any of the current monitors to run. They all fail to start

[ceph-users] Re: Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

Hi, For this upgrade you need at least some mon's up, then you can redeploy your pacific mon's to rocksdb k Sent from my iPhone > On Oct 10, 2023, at 02:01, Waywatcher wrote: > > I upgraded my CEPH cluster without properly following the mon upgrade so > they were no longer on leveldb. > > P

[ceph-users] Re: replacing storage server host (not drives)

Hi, The procedure is simple: get another host and put current disk to new host. Setup boot and network's and back to business k Sent from my iPhone > On Sep 26, 2023, at 17:38, Wyll Ingersoll > wrote: > > What is the recommended procedure for replacing the host itself without > destroying

[ceph-users] Re: cephfs mount 'stalls'

Hi, William, this is fuse client, not the kernel Mark, you can use kernel client. Stock c7 or install, for example, kernel-ml from ELrepo [1], and use the latest krbd version [1] https://elrepo.org/tiki/HomePage#elrepo-kernel k Sent from my iPhone > On 17 Sep 2023, at 22:10, William Edwards

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi Igor, > On 12 Sep 2023, at 15:28, Igor Fedotov wrote: > > Default hybrid allocator (as well as AVL one it's based on) could take > dramatically long time to allocate pretty large (hundreds of MBs) 64K-aligned > chunks for BlueFS. At the original cluster it was exposed as 20-30 sec OSD > st

[ceph-users] Re: Rocksdb compaction and OSD timeout

This cluster use the default settings or something for Bluestore was changed? You can check this via `ceph config diff` As Mark said, it will be nice to have a tracker, if this really release problem Thanks, k Sent from my iPhone > On 7 Sep 2023, at 20:22, J-P Methot wrote: > > We went from

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi, > On 7 Sep 2023, at 18:21, J-P Methot wrote: > > Since my post, we've been speaking with a member of the Ceph dev team. He > did, at first, believe it was an issue linked to the common performance > degradation after huge deletes operation. So we did do offline compactions on > all our OS

[ceph-users] Re: Rocksdb compaction and OSD timeout

Hi, > On 7 Sep 2023, at 10:05, J-P Methot wrote: > > We're running latest Pacific on our production cluster and we've been seeing > the dreaded 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after > 15.00954s' error. We have reasons to believe this happens each time the > RocksDB co

[ceph-users] Re: Status of diskprediction MGR module?

Hi, > On 28 Aug 2023, at 12:45, Robert Sander wrote: > > Several years ago the diskprediction module was added to the MGR collecting > SMART data from the OSDs. > > There were local and cloud modes available claiming different accuracies. Now > only the local mode remains. > > What is the cu

[ceph-users] Re: User + Dev Monthly Meeting Minutes 2023-08-24

On 24 Aug 2023, at 18:51, Laura Flores wrote: > > Need pacific backport for https://tracker.ceph.com/issues/59478 > > - Laura will check on this, although a Pacific backport is unlikely due > to incompatibilities from the scrub backend refactoring. Laura, this fix "for malformed fix" of ear

[ceph-users] Re: [quincy] Migrating ceph cluster to new network, bind OSDs to multple public_nework

Hi, This how OSD's woks. For change the network subnet you need to setup reachability of both: old and new network, until end of migration k Sent from my iPhone > On 22 Aug 2023, at 10:43, Boris Behrens wrote: > > The OSDs are still only bound to one IP address. __

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

Hi, > On 17 Aug 2023, at 18:21, yosr.kchao...@gmail.com wrote: > > Thanks for your reply. By the Bluestore compression I mean the compression > enabled on the pool level. It is also called inline compression. > https://docs.ceph.com/en/reef/rados/configuration/bluestore-config-ref/#inline-compre

[ceph-users] Re: Check allocated RGW bucket/object size after enabling Bluestore compression

Hi, What you mean, Bluestore compression? The rgw compression is a rados compression, not the compress by rgw itself. You can setup different storage classes and upload to same pool uncompressed, or compressed objects The compression ratio you can determine with exporter [1] [1] https://githu

[ceph-users] Re: Can't join new mon - lossy channel, failing

> On 16 Aug 2023, at 13:23, Josef Johansson wrote: > > I'm running ceph version 15.2.16 (a6b69e817d6c9e6f02d0a7ac3043ba9cdbda1bdf) > octopus (stable), that would mean I am not running the fix. > > Glad to know that an upgrade will solve the issue! I'm not 100% sure that this tracker, exactly

[ceph-users] Re: Can't join new mon - lossy channel, failing

Hi, > On 16 Aug 2023, at 11:30, Josef Johansson wrote: > > Let's do some serious necromancy here. > > I just had this exact problem. Turns out that after rebooting all nodes (one > at the time of course), the monitor could join perfectly. > > Why? You tell me. We did not see any traces of the

[ceph-users] Re: v18.2.0 Reef released

Hi, Thanks for the release! Please, upgrade the OS Platform docs, currently there lack of Reef in ABC tests: https://tracker.ceph.com/issues/62354 Thanks, k > On 7 Aug 2023, at 21:37, Yuri Weinstein wrote: > > We're very happy to announce the first stable release of the Reef series. > > We

[ceph-users] Re: Ceph Quincy and liburing.so.2 on Rocky Linux 9

Hi, In most cases the 'Alternative' distro like Alma or Rocky have outdated versions of packages, if we compared it with CentOS Stream 8 or CentOS Stream 9. For example is a golang package, on c8s is a 1.20 version on Alma still 1.19 You can try to use c8s/c9s or try to contribute to your distr

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

Hi, Can you show `smartctl -a` for this device? This drives show input/output errors in dmesg when you try to run ceph-osd? k Sent from my iPhone > On 2 Aug 2023, at 21:44, Greg O'Neill wrote: > > Syslog says the drive is not in write-protect mode, however smart says life > remaining is at 1

[ceph-users] Re: Luminous Bluestore issues and RGW Multi-site Recovery

Hi > > On 31 Jul 2023, at 03:12, Gregory O'Neill wrote: > > /build/ceph-12.2.12/src/os/bluestore/KernelDevice.cc: 397: FAILED assert(0 > == "unexpected aio error") The Kernel was returned error actually. Try look to dmesg, may be your SATA SSD in read only mode k Sent from my iPhone ___

[ceph-users] Re: cephadm and kernel memory usage

Without determining what exactly process (kernel or userspace) "eat" memory, the ceph-users can't tell what exactly use memory, because don't see your display with your eyes 🙂 You should run this commands on good & bad hosts to see the real difference. This may be related to kernel version, or

[ceph-users] Re: cephadm and kernel memory usage

Good, > On 24 Jul 2023, at 20:01, Luis Domingues wrote: > > Of course: > > free -h > totalusedfree shared buff/cache available > Mem: 125Gi96Gi 9.8Gi 4.0Gi19Gi > 7.6Gi > Swap:0B 0B 0B

[ceph-users] Re: Does ceph permit the definition of new classes?

Hi, You definitely can add any other class name k Sent from my iPhone > On 24 Jul 2023, at 16:04, wodel youchi wrote: > > Can I define new device classes in ceph, I know that there are hdd, ssd and > nvme, but can I define other classes? ___ ceph-use

[ceph-users] Re: cephadm and kernel memory usage

Hi, Can you paste `free -h` output for this hosts? k Sent from my iPhone > On 24 Jul 2023, at 14:42, Luis Domingues wrote: > > Hi, > > So after, looking into OSDs memory usage, which seem to be fine, on a > v16.2.13 running with cephadm, on el8, it seems that the kernel is using a > lot o

[ceph-users] Re: librbd hangs during large backfill

Hi, Check you libvirt limits for qemu open files/sockets. Seems, when you added new OSD's, your librbd client limit reached k Sent from my iPhone > On 18 Jul 2023, at 19:32, Wesley Dillingham wrote: > > Did your automation / process allow for stalls in between changes to allow > peering to

[ceph-users] Re: Another Pacific point release?

Hi, > On 17 Jul 2023, at 12:53, Ponnuvel Palaniyappan wrote: > > The typical EOL date (2023-06-01) has already passed for Pacific. Just > wondering if there's going to be another Pacific point release (16.2.14) in > the pipeline. Good point! At least, for possibility upgrade RBD clusters from N

[ceph-users] Re: MON sync time depends on outage duration

This is a guess, the databases is like to swell. Especially the Level DB's, can grow x2 and reduce tens of percent of total size. This may be just another SST file creation, 1GB by default, Ii I remember it right Do you was looks to Grafana, about this HDD's utilization, IOPS? k Sent from my iP

[ceph-users] Re: MON sync time depends on outage duration

Hi, And additional to Dan suggestion, the HDD is not a good choices for RocksDB, which is most likely the reason for this thread, I think that from the 3rd time the database just goes into compaction maintenance k Sent from my iPhone > On 6 Jul 2023, at 23:48, Eugen Block wrote: > The MON st

[ceph-users] Re: ceph quota qustion

Hi, This is incomplete multiparts I guess, you should remove it first. Don't know how S3 Browser works with this entities k Sent from my iPhone > On 6 Jul 2023, at 07:57, sejun21@samsung.com wrote: > > Hi, I contact you for some question about quota. > > Situation is following below. >

[ceph-users] Re: Get bucket placement target

Hi, > On 3 Jul 2023, at 12:23, mahnoosh shahidi wrote: > > So clients can not get the value which they set in the LocationConstraint > field in the create bucket request as in this doc > ? LocationConstraint in this case is

[ceph-users] Re: Get bucket placement target

Hi, > On 2 Jul 2023, at 17:17, mahnoosh shahidi wrote: > > Is there any way for clients (without rgw-admin access) to get the > placement target of their S3 buckets? The "GetBucketLocation'' api returns > "default" for all placement targets and I couldn't find any other S3 api > for this purpose

[ceph-users] Re: Applying crush rule to existing live pool

Hi, > On 27 Jun 2023, at 18:31, Rok Jaklič wrote: > > I want to place an existing pool with data to ssd-s. > > I've created crush rule: > ceph osd crush rule create-replicated replicated_ssd default host ssd > > If I apply this rule to the existing pool default.rgw.buckets.index with > 180G of

[ceph-users] Re: rbd ls failed with operation not permitted

Hi, > On 7 Jun 2023, at 14:39, zyz wrote: > > When set the user's auth and then ls namespace, it is ok. > > > But when I set the user's auth with namespace, ls namespace returns with > error, but why? Because data with namespaces in "without namespace" space k _

  1   2   3   4   5   >