This is also the case with us for Cephfs clients. We use the kernel mount, but
still you need to install ceph-common. Which isn't possible with the latest
version of Reef and thus those are stuck on 18.2.4.
Or am I doing something wrong?
> Op 03-06-2025 06:03 CEST schreef Dmitry Melekhov :
>
>
openssl-libs.x86_64
1:3.2.2-6.el9_5.1
@rhel-9-for-x86_64-baseos-rpms
> Op 09-05-2025 21:34 CEST schreef Sake Ceph :
>
>
> We're running upd
We're running update to date RHEL 9 machines.. They're offline, so it could be
we need to sync the central repo, but I don't think this version is available.
> Op 09-05-2025 21:12 CEST schreef Konstantin Shalygin :
>
>
> Hi,
>
> > On 8 May 2025, at 23:12, Erwin Bogaard wrote:
> >
> > It loo
Yup, also my end...
- nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit)
> Op 08-05-2025 22:12 CEST schreef Erwin Bogaard :
>
>
> It looks like there is an issue with the package-based 18.2.7 release, when
> upgrading from 18.2.6 on el9.
> There seems to be a new (unfulfilled) dependency tha
we will backport the alloy thing to squid (could break
> stuffs), but we can say more when we test it.
>
> And as for grafana it'll be 11.6.0 which is what we will use:
> https://github.com/ceph/ceph/pull/62827
>
> Regards,
> Nizam
>
>
> On Fri, Apr 25, 2025 at
t to target for “stable” releases for the moment.
>
>
> --
> Dan Brown
>
> > On Apr 18, 2025, at 10:47, Sake Ceph wrote:
> >
> > Really great you moved to the default grafana image and moving the
> > customization to cephadm. This makes upgrading ind
>
> sudo ceph cephadm -v registry-login --registry-json login
>
> or only
>
> sudo cephadm -v registry-login --registry-json login
>
> to login only the host where you issued that command? But since you
> mention that all hosts have already pulled the image it's not t
and
> 'addr'
>
> exit maint 2/3
> Ceph cluster f3e63d9e-2f4c-11ef-87a2-0f1170f55ed5 on cephbackup-osd1 has
> exited maintenance mode
> exit maint 3/3
> Error EINVAL: Host cephbackup-osd1 is not in maintenance mode
> Fri Apr 25 07:17:58 CEST 2025 cluster state is HE
t;
>
> Thanks.
>
>
> -- Michael
>
>
>
>
> Get Outlook for Android (https://aka.ms/AAb9ysg)
> --
> From: Sake Ceph
> Sent: Friday, June 14, 2024 4:28:34 AM
> To: Michael Worsham ; ceph-users@ceph.io
>
> Subject: Re:
gin --registry-json login'? Because the first would login
> only the local node into the registry, the latter does it for all
> nodes in the host list.
>
> Zitat von Sake Ceph :
>
> > We're used a local registry (docker registry), but needed to switch
>
We're used a local registry (docker registry), but needed to switch to Harbor.
The old registry needed credentials and was working fine.
I used "sudo cephadm -v registry-login --registry-json login" to login to the
new registry. login is the json with the credentials.
When starting the upgrade,
l repos.
>
>
> On Thu, Apr 24, 2025 at 10:15 AM Sake Ceph wrote:
> > We're used a local registry (docker registry), but needed to switch to
> > Harbor. The old registry needed credentials and was working fine.
> >
> > I used "sudo cephadm -v registry-log
t; Will check both of them next week after the holidays.
>
> [1] https://docs.ceph.com/en/latest/releases/squid/#monitoring
>
> Regards,
> Nizam
>
>
> On Thu, Apr 17, 2025 at 7:43 PM Sake Ceph wrote:
> > Squid is already on 10.4.0 and it looks like it using t
/samba.org/samba-server:devbuilds-centos-amd64',
> 'samba')
> SAMBA_METRICS = _create_image('quay.io/samba.org/samba-metrics:latest',
> 'samba_metrics')
> NGINX = _create_image('quay.io/ceph/nginx:sclorg-nginx-126', 'nginx')
&g
But Grafana 9.4 is EOL for a long time. Shouldn't it be time to upgrade the
image?
Kind regards,
Sake
> Op 17-04-2025 09:14 CEST schreef Robert Sander :
>
>
> Hi,
>
> Am 4/16/25 um 21:11 schrieb Anthony D'Atri:
> > This is covered in the docs:
> >
> > https://docs.ceph.com/en/reef/cephadm/se
I stumbled on this problem earlier, port 9926 isn't being opened. See also
thread "Grafana dashboards is missing data".
A tracker is already opened to fix the issue:
https://tracker.ceph.com/issues/67975
> Op 25-11-2024 13:44 CET schreef Kilian Ries :
>
>
> Prometheus metrics seem to be broke
ay be able to do this, but I don't think
> > cephadm does.
> > Adam, is there some way to finagle this with pod placement rules (ie,
> > tagging nodes as mds and mds-standby, and then assigning special mds config
> > info to corresponding pods)?
> > -Greg
> >
gt;
> You'll have to monitor each MDS state and restart any non-local active MDSs
> to reverse roles.
>
> Regards,
> Frédéric.
>
> - Le 29 Oct 24, à 14:06, Sake Ceph c...@paulusma.eu a écrit :
>
> > Hi all
> > We deployed successfully a stretched clus
Hi all
We deployed successfully a stretched cluster and all is working fine. But is it
possible to assign the active MDS services in one DC and the standby-replay in
the other?
We're running 18.2.4, deployed via cephadm. Using 4 MDS servers with 2 active
MDS on pinnend ranks and 2 in standby-re
;
> (http://quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4fe00ed30cb1c1f91532926db20d3c')],
> 'rank': None, 'rank_generation': None,
>
>
> I opened the following tracker to fix the issue:
> https://tracker.ceph.com/issues/6797
'.ports'
> /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta
> []
>
> soc9-ceph:~ # jq '.ports' /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta
> [
>3000
> ]
>
> But that's about all I can provide here. Maybe the host OS plays some
&
ceph orch ls ceph-exporter --export
> service_type: ceph-exporter
> service_name: ceph-exporter
> placement:
>host_pattern: '*'
> spec:
>prio_limit: 5
>stats_period: 5
>
> Zitat von Sake Ceph :
>
> > Hello Eugen,
> >
> > Well noth
e is not available
>
> Zitat von Eugen Block :
>
> > Do you see anything in the cephadm.log related to the firewall?
> >
> > Zitat von Sake Ceph :
> >
> >> After opening port 9926 manually, the Grafana dashboards show the data.
> >> So is this a bug?
After opening port 9926 manually, the Grafana dashboards show the data.
So is this a bug?
Kind regards,
Sake
> Op 06-09-2024 17:39 CEST schreef Sake Ceph :
>
>
> That is working, but I noticed the firewall isn't opened for that port.
> Shouldn't cephadm manage this,
That is working, but I noticed the firewall isn't opened for that port.
Shouldn't cephadm manage this, like it does for all the other ports?
Kind regards,
Sake
> Op 06-09-2024 16:14 CEST schreef Björn Lässig :
>
>
> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrie
; As a workaround you can use: ceph config set mgr
> mgr/prometheus/exclude_perf_counters false
>
> However I understand that deploying a ceph-exporter daemon on each host is
> the proper fix. You may still be missing some configuration for it?
>
> On Thu, 5 Sept 2024 at 08:25
his pull request will fix your problem:
> https://github.com/ceph/ceph/pull/58402
>
> Regards,
>
> Frank
>
> Sake Ceph wrote:
> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For
> > example the Osd latency under OSD device details
After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are empty. For example
the Osd latency under OSD device details or the Osd Overview has a lot of No
data messages.
I deployed ceph-exporter on all hosts, am I missing something? Did even a
redeploy of prometheus.
Kind regards,
Sake
But the client which is doing the rsync, doesn't hold any caps after the rsync.
Cephfs-top shows 0 caps. Even a system reboot of the client doesn't make a
change.
Kind regards,
Sake
> Op 03-09-2024 04:01 CEST schreef Alexander Patrakov :
>
>
> MDS cannot release an inode if a client has ca
o reduce the impact of the MDS slowed down by
> > excessive memory consumption, I would suggest installing earlyoom,
> > disabling swap, and configuring earlyoom as follows (usually through
> > /etc/sysconfig/earlyoom, but could be in a different place on your
> > distribu
> Got it.
>
> However, to narrow down the issue, I suggest that you test whether it
> still exists after the following changes:
>
> 1. Reduce max_mds to 1.
> 2. Do not reduce max_mds to 1, but migrate all clients from a direct
> CephFS mount to NFS.
>
> On Sat, A
rakov :
>
>
> Got it.
>
> However, to narrow down the issue, I suggest that you test whether it
> still exists after the following changes:
>
> 1. Reduce max_mds to 1.
> 2. Do not reduce max_mds to 1, but migrate all clients from a direct
> CephFS mount to NFS.
>
>
know about.
>
> On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph wrote:
> >
> > @Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be
> > used, but this doesn't change the problem.
> >
> > @Alexander: the problem is in the active
nderlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit
some issues with automatically upgrading hosts.
Kind regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
screenshots, not sure how to share.
Kind regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
What I read on the Slack channel is that the publication job got stuck late in
the day and the restart finished late. I'll guess they announce today the new
version.
Kind regards,
Sake
> Op 24-07-2024 13:05 CEST schreef Alfredo Rezinovsky :
>
>
> Ceph dashboard offers me to upgrade to v18.
Not yet released. Every x.1.z release is release candidate. Always wait for the
x.2.z release (in this case 19.2.0) and the official release notes on
docs.ceph.com :-)
> Op 21-07-2024 18:32 CEST schreef Nicola Mori :
>
>
> Dear Ceph users,
>
> on quay.io I see available images for 19.1.0. Y
Edit: someone made some changes which broke some tasks when selecting the
cephadm host to use. Just keep in mind it's an example
> Op 14-06-2024 10:28 CEST schreef Sake Ceph :
>
>
> I needed to do some cleaning before I could share this :)
> Maybe you or someone else ca
his.
>
> -- Michael
> ____
> From: Sake Ceph
> Sent: Thursday, June 13, 2024 4:05 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] Re: Patching Ceph cluster
>
> This is an external email. Please take care when clicking links or opening
> attachments. When in doubt,
Yeah we fully automated this with Ansible. In short we do the following.
1. Check if cluster is healthy before continuing (via REST-API) only health_ok
is good
2. Disable scrub and deep-scrub
3. Update all applications on all the hosts in the cluster
4. For every host, one by one, do the followi
arting the containers. The start up fails because it can't pull
the container image because authentication is required (our instance is offline
and we're using a local image registry with authentication).
Kind regards,
Sake
> Op 04-06-2024 14:40 CEST schreef Robert Sander :
&g
y has some advice I would gladly hear about it!
KIND regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Hi Xiubo
Thank you for the explanation! This won't be a issue for us, but made me think
twice :)
Kind regards,
Sake
> Op 04-06-2024 12:30 CEST schreef Xiubo Li :
>
>
> On 6/4/24 15:20, Sake Ceph wrote:
> > Hi,
> >
> > A little break into this thread, b
Hi,
A little break into this thread, but I have some questions:
* How does this happen, that the filesystem gets into readonly modus
* Is this avoidable?
* How-to fix the issue, because I didn't see a workaround in the mentioned
tracker (or I missed it)
* With this bug around, should you use c
Hi
Isn't this just the limit of one HDD or the other HDD's for providing the data?
Don't forget, recovery will drop even more for the last few objects. At least I
noticed this when replacing a drive in my (little) cluster.
Kind regards,
Sake
> Op 26-05-2024 09:36 CEST schreef Mazzystr :
>
e.slack.com/archives/C054Q1NUBQT/p1711041666180929
>
> Regards
> YuriW
>
> On Thu, May 23, 2024 at 6:22 AM Sake Ceph wrote:
> >
> > I was wondering what happened to the release of 18.2.3? Validation started
> > on April 13th and as far as I know there have been a cou
but I really need some
fixes of this release.
Kind regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
hopes it will have enough RAM to complete the replay?
> >
> > On 4/22/24 11:37 AM, Sake Ceph wrote:
> >> Just a question: is it possible to block or disable all clients? Just
> >> to prevent load on the system.
> >>
> >> Kind regard
Just a question: is it possible to block or disable all clients? Just to
prevent load on the system.
Kind regards,
Sake
> Op 22-04-2024 20:33 CEST schreef Erich Weiler :
>
>
> I also see this from 'ceph health detail':
>
> # ceph health detail
> HEALTH_WARN 1 filesystem is degraded; 1 MDSs
Hi Matthew,
Cephadm doesn't cleanup old container images, at least with Quincy. After a
upgrade we run the following commands:
sudo podman system prune -a -f
sudo podman volume prune -f
But if someone has a better advice, please tell us.
Kind regards,
Sake
> Op 19-04-2024 10:24 CEST schreef
I would say drop it for squid release or if you keep it in squid, but going to
disable it in a minor release later, please make a note in the release notes if
the option is being removed.
Just my 2 cents :)
Best regards,
Sake
___
ceph-users mailing
support from dashboard because of security
> reasons. (But so far we are planning to keep it as it is atleast for the
> older releases)
>
> Regards,
> Nizam
>
>
> On Thu, Jan 25, 2024, 19:41 Sake Ceph wrote:
> > After upgrading to 17.2.7 our load balancers can'
After upgrading to 17.2.7 our load balancers can't check the status of the
manager nodes for the dashboard. After some troubleshooting I noticed only TLS
1.3 is availalbe for the dashboard.
Looking at the source (quincy), TLS config got changed from 1.2 to 1.3.
Searching in the tracker I found
es/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01
rm: cannot remove
'/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01':
Input/output error
Best regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
023 13:43 CET schreef Sake Ceph :
>
>
> Hi!
>
> As I'm reading through the documentation about subtree pinning, I was
> wondering if the following is possible.
>
> We've got the following directory structure.
> /
> /app1
> /app2
> /app3
>
d /app4 to rank 3?
I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers.
Best regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
That wasn't really clear in the docs :(
> Op 21-12-2023 17:26 CET schreef Patrick Donnelly :
>
>
> On Thu, Dec 21, 2023 at 3:05 AM Sake Ceph wrote:
> >
> > Hi David
> >
> > Reducing max_mds didn't work. So I executed a fs reset:
> > c
ph fs reset atlassian-prod --yes-i-really-mean-it
This brought the fs back online and the servers/applications are working again.
Question: can I increase the max_mds and active standby_replay?
Will collect logs, maybe we can pinpoint the cause.
Best regards,
583} state up:resolve seq 571
join_fscid=2 addr
[v2:10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat
{c=[1],r=[1],i=[7ff]}]
Best regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to cep
Don't forget with stretch mode, osds only communicate with mons in the same DC
and the tiebreaker only communicate with the other mons (to prevent split brain
scenarios).
Little late response, but I wanted you to know this :)
___
ceph-users mailing lis
gt; stack trace the complicated password doesn't seem to be applied
> > (don't know why yet). But since it's an "initial" password you can
> > choose something simple like "admin", and during the first login you
> > are asked to change it
I believe they are working on it or want to work on it to revert from a
stretched cluster, because of the reason you mention: if the other datacenter
is totally burned down, you maybe want for the time being switch to one
datacenter setup.
Best regards,
Sake
> Op 09-11-2023 11:18 CET schreef
I tried everything at this point, even waited a hour, still no luck. Got it 1
time accidentally working, but with a placeholder for a password. Tried with
correct password, nothing and trying again with the placeholder didn't work
anymore.
So I thought to switch the manager, maybe something is
..
Found those files with 'find / -name *grafana*'.
> Op 09-11-2023 09:53 CET schreef Eugen Block :
>
>
> What doesn't work exactly? For me it did...
>
> Zitat von Sake Ceph :
>
> > To bad, that doesn't work :(
> >> Op 09-11-2023 09:07 CET s
To bad, that doesn't work :(
> Op 09-11-2023 09:07 CET schreef Sake Ceph :
>
>
> Hi,
>
> Well to get promtail working with Loki, you need to setup a password in
> Grafana.
> But promtail wasn't working with the 17.2.6 release, the URL was set to
> co
ng for. Not sure what this has to do with Loki though.
>
> Eugen
>
> Zitat von Sake Ceph :
>
> > I configured a password for Grafana because I want to use Loki. I
> > used the spec parameter initial_admin_password and this works fine for a
> > staging environ
credentials error
on environment where I tried to use Grafana with Loki in the past (with 17.2.6
of Ceph/cephadm). I changed the password in the past within Grafana, but how
can I overwrite this now? Or is there a way to cleanup all Grafana files?
Best regards,
Sake
regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Thank you Patrick for responding and fix the issue! Good to know the issue is
know and been worked on :-)
> Op 21-07-2023 15:59 CEST schreef Patrick Donnelly :
>
>
> Hello Sake,
>
> On Fri, Jul 21, 2023 at 3:43 AM Sake Ceph wrote:
> >
> > At 01:27 this morn
large
(15GB/9GB); 0 inodes in use by clients, 0 stray files
=== Full health status ===
[WARN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache
mds.atlassian-prod.mds4.qlvypn(mds.0): MDS cache is too large
(15GB/9GB); 0 inodes in use by clients, 0 stray files
Best regards,
er bug
report?
And does someone know a workaround to set the correct URL for the time being?
Best regards,
Sake
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
71 matches
Mail list logo
