[ceph-users] CephFS 16.2.10 problem
Good evening! The following problem occurred. There is a cluster ceph 16.2.10 The cluster was operating normally on Friday. Shut down cluster: -Excluded all clients Executed commands: ceph osd set noout ceph osd set nobackfill ceph osd set norecover ceph osd set norebalance ceph osd set nodown ceph osd set pause Turned off the cluster, checked server maintenance. Enabled cluster. He gathered himself, found all the nodes, and here the problem began. After all OSD went up and all pg became available, cephfs refused to start. Now mds are in the replay status, and do not go to the ready status. Previously, one of them was in the replay (laggy) status, but we executed command: ceph config set mds mds_wipe_sessions true After that, mds switched to the status of replays, the third in standby status started, and mds crashes with an error stopped. But cephfs is still unavailable. What else can we do? The cluster is very large, almost 200 million files. Best regards A.Tsivinsky e-mail: alexey.tsivin...@baikalelectronics.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS 16.2.10 problem
> > The following problem occurred. > There is a cluster ceph 16.2.10 > The cluster was operating normally on Friday. Shut down cluster: > -Excluded all clients > Executed commands: > ceph osd set noout > ceph osd set nobackfill > ceph osd set norecover > ceph osd set norebalance > ceph osd set nodown > ceph osd set pause > Turned off the cluster, checked server maintenance. > Enabled cluster. He gathered himself, found all the nodes, and here the > problem began. After all OSD went up and all pg became available, cephfs > refused to start. > Now mds are in the replay status, and do not go to the ready status. > Previously, one of them was in the replay (laggy) status, but we > executed command: ceph config set mds mds_wipe_sessions true > After that, mds switched to the status of replays, the third in standby > status started, and mds crashes with an error stopped. > But cephfs is still unavailable. > What else can we do? > The cluster is very large, almost 200 million files. > I assume you tried to start just one mds and wait until it would come up as active (before starting the others)? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Migrated to cephadm, rgw logs to file even when rgw_ops_log_rados is true
Hi!
Currently I have limitted the optput of rgw log to syslog from rsyslog (as
suggested by
Anthony), limitted docker logs from daemon.json.
I still get ops logs written to both logs pool and ops log file
(ops-log-ceph-client.rgw.hostname.log).
How to stop logging ops log on rgw disk and keep logs on logs pool?
Current config:
globaladvanced debug_rgw
0/0
globaladvanced
rgw_enable_ops_log true
globaladvanced
rgw_ops_log_rados true
Thank you!
--
Paul Jurco
On Fri, Nov 22, 2024 at 6:11 PM Paul JURCO wrote:
> Hi,
> we recently migrated to cephadm from ceph-deploy a 18.2.2 ceph cluster
> (Ubuntu with docker).
> RGWs are separate vms.
> We noticed syslog increased a lot due to rgw's access logs sent to it.
> And because we use to log ops, a huge ops log file on
> /var/log/ceph/cluster-id/ops-log-ceph-client.rgw.hostname-here.log.
>
> While having "rgw_ops_log_rados": "true", oplogs goes to both file and
> rados pool for logs.
> If false it doesn't log anything, as expected.
> How to stop dockered rgws to log to syslog and to a file on disk, but to
> keep opslog in logs pool?
>
> Config is:
> globalbasic
> log_to_journald false
> globaladvanced
> rgw_enable_ops_log false
> globaladvanced
> rgw_ops_log_radostrue
>
> A few hours later after after enabling it back, after massive cleanup, it
> does logging ops but only to files.
> How to get ops logs in rados pool and access log to a file on disk but not
> on syslog?
> I have add this to daemon.json to limit access logs to accumulate on
> /var/log/docker/containers/rand/rand/json.log file:
>
> {
> "log-driver": "local",
> "log-opts": {
> "max-size": "512m",
> "max-file": "3"
> }
> }
>
>
> Thank you!
> Paul
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Separate gateway for bucket lifecycle
Hi, just remove them from balancer. Also, there are two configs you want to be true on the rgws for LC and GC processing and false on the rgws that are exposed to clients: rgw_enable_lc_threads = true rgw_enable_gc_threads = true -- Paul On Mon, Nov 25, 2024 at 8:40 AM Szabo, Istvan (Agoda) < istvan.sz...@agoda.com> wrote: > Hi, > > I remember some discussion that someone is using separated gateways for > bucket lifecycles but I couldn't find it. > > How is that can be possible? > > Thank you > > > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by copyright > or other legal rules. If you have received it by mistake please let us know > by reply email and delete it from your system. It is prohibited to copy > this message or disclose its content to anyone. Any confidentiality or > privilege is not waived or lost by any mistaken delivery or unauthorized > disclosure of the message. All messages sent to and from Agoda may be > monitored to ensure compliance with company policies, to protect the > company's interests and to remove potential malware. Electronic messages > may be intercepted, amended, lost or deleted, or contain viruses. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: CephFS 16.2.10 problem
Good afternoon We tried to leave only one mds, stopped others, even deleted one, and turned off the requirement for stand-by mds. Nothing helped, mds remained in the status of replays. Current situation: we now have two active mds in the status of replays, and one in stand-by. At the same time, in the logs we see a message mds.0.660178 waiting for osdmap 123138 (which blocklists prior instance) At the same time, there is no activity on both mds. The launch of the cephfs-journal-tool journal inspect utility does not produce any results - the utility worked for 12 hours and did not produce anything, we stopped it. Maybe the problem is this blocking? How to remove it? Best regards! Alexey Tsivinsky e-mail: a.tsivin...@baikalelectronics.com От: Marc Отправлено: 25 ноября 2024 г. 1:47 Кому: Цивинский Алексей Александрович; ceph-users@ceph.io Тема: RE: CephFS 16.2.10 problem > > The following problem occurred. > There is a cluster ceph 16.2.10 > The cluster was operating normally on Friday. Shut down cluster: > -Excluded all clients > Executed commands: > ceph osd set noout > ceph osd set nobackfill > ceph osd set norecover > ceph osd set norebalance > ceph osd set nodown > ceph osd set pause > Turned off the cluster, checked server maintenance. > Enabled cluster. He gathered himself, found all the nodes, and here the > problem began. After all OSD went up and all pg became available, cephfs > refused to start. > Now mds are in the replay status, and do not go to the ready status. > Previously, one of them was in the replay (laggy) status, but we > executed command: ceph config set mds mds_wipe_sessions true > After that, mds switched to the status of replays, the third in standby > status started, and mds crashes with an error stopped. > But cephfs is still unavailable. > What else can we do? > The cluster is very large, almost 200 million files. > I assume you tried to start just one mds and wait until it would come up as active (before starting the others)? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Separate gateway for bucket lifecycle
Hi, I remember some discussion that someone is using separated gateways for bucket lifecycles but I couldn't find it. How is that can be possible? Thank you This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
