[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-11 Thread Alex
Sounds good to me. I responded to your comment in the PR. Thanks. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-11 Thread John Mulligan
On Thursday, April 10, 2025 10:42:50 PM Eastern Daylight Time Alex wrote: > I made a Pull Request for cephadm.log set DEBUG. > Not sure if I should merge it. Please, no. Even if github allows you to (I think it won't) you should not merge your own PRs unless you are a component lead and it is an

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Anthony D'Atri
Link please. > On Apr 10, 2025, at 10:59 PM, Alex wrote: > > I made a Pull Request for cephadm.log set DEBUG. > Not sure if I should merge it. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ce

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Alex
I made a Pull Request for cephadm.log set DEBUG. Not sure if I should merge it. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Eugen Block
That was my assumption, yes. Zitat von Alex : Is this bit of code responsible for hardcoding DEBUG to cephadm.log? 'loggers': { '': { 'level': 'DEBUG', 'handlers': ['console', 'log_file'], } } in /var/lib/ceph//cephadm.* ? ___

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Alex
I think it's the same block of code Eugen found. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Laimis Juzeliūnas
Hey all, Just confirming that the same debug level has been in Reef and Squid. We got so used to it that just decided not to care anymore. Best, Laimis J. > On 8 Apr 2025, at 14:21, Alex wrote: > > Interesting. So it's like that for everybody? > Meaning cephadm.log logs debug messages. >

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Alex
Is this bit of code responsible for hardcoding DEBUG to cephadm.log? 'loggers': { '': { 'level': 'DEBUG', 'handlers': ['console', 'log_file'], } } in /var/lib/ceph//cephadm.* ? ___ ceph-users mailing list

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Alex
I did have to add "su root root" to the log rotate script to fix the permissions issue. There's a RH KB article and Ceph github pull requests to fix it. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@cep

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-10 Thread Alex
Thanks! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Eugen Block
This is from a openSUSE base OS: # cat /etc/logrotate.d/cephadm # created by cephadm /var/log/ceph/cephadm.log { rotate 7 daily compress missingok notifempty su root root } Zitat von Alex : I did have to add "su root root" to the log rotate script to fix the permissio

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Alex
How does that work? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Eugen Block
You would have to do that after each upgrade. Seems easier to just accept the debug logs. ;-) Zitat von Alex : Official IBM and RH "fix" is to replace DEBUG with INFO in /var/lib/ceph//cephadm.hash ¯\_ (ツ) _/¯ ___ ceph-users mailing list -- ceph

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Anthony D'Atri
I have at times used admin sockets to tell daemons to re-open log files after running a truncate > On Apr 9, 2025, at 3:51 PM, Eugen Block wrote: > > I haven't modified the logrotate config from the cephadm package. There's no > cephadm process running, it's invoked everytime the orchestrator

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Alex
Thanks. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Eugen Block
I haven't modified the logrotate config from the cephadm package. There's no cephadm process running, it's invoked everytime the orchestrator does one of its checks (host, network, osd specs etc.). So I don't see a reason to tweak anything here, I stick to the defaults. Zitat von Alex : H

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Alex
Haha yeah think that's what we're doing. I'm just going to add it to logrotate. Do you use copytruncate option or postrotate to restart ceph? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Alex
Thanks Eugen! I think you're right since support had me grep for the same code. Seems crazy though that it's hardcoded doesn't it? I guess we can mod the Python file, but you'd think that wouldn't be necessary. Should we make a feature request or modify the code ourselves and make a pull request?

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-09 Thread Alex
Official IBM and RH "fix" is to replace DEBUG with INFO in /var/lib/ceph//cephadm.hash ¯\_ (ツ) _/¯ ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Alex
I mean this bit 'log_file': { 'level': 'DEBUG', 'class': 'logging.handlers.WatchedFileHandler', 'formatter': 'cephadm', 'filename': '%s/cephadm.log' % LOG_DIR, } ___ ceph-users mailing list

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Alex
What about Pacific and Quincy? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Alex
Can someone paste in here their copy of logrotate? The trick always with rotating logs is that the service writing to it needs to be restarted or told to stop writing so the file handle gets closed. Otherwise it stays open and the free disc space isn't recovered. ___

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Eugen Block
I guess it's debatable as almost everything. ;-) One of the advantages is that you usually see immediately what is failing, you don't have to turn on debug first, retry the deployment or whatever again to reproduce. The file doesn't really grow to huge sizes (around 2 MB per day or so) and g

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Eugen Block
It's the same across all versions I have used so far. Zitat von Alex : What about Pacific and Quincy? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Alex
Awesome. Wish I knew that before I spent half a day trying to overwrite it. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Alex
Interesting. So it's like that for everybody? Meaning cephadm.log logs debug messages. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log

2025-04-08 Thread Eugen Block
I think it's just hard-coded in the cephadm binary [0]: _log_file_handler = { 'level': 'DEBUG', 'class': 'logging.handlers.WatchedFileHandler', 'formatter': 'cephadm', 'filename': '%s/cephadm.log' % LOG_DIR, } I haven't looked to deep yet if it can be overridden, but didn't find