The lack of warnings in ceph -w for this issue is a bug in Emperor. It's resolved in Firefly. -Greg
On Wed, Jun 18, 2014 at 3:49 AM, Andrija Panic <andrija.pa...@gmail.com> wrote: > > Hi Gregory, > > indeed - I still have warnings about 20% free space on CS3 server, where MON > lives...strange is that I don't get these warnings with prolonged "ceph -w" > output... > [root@cs2 ~]# ceph health detail > HEALTH_WARN > mon.cs3 addr 10.44.xxx.12:6789/0 has 20% avail disk space -- low disk space! > > I don't understand, how is this possible to get warnings - I have folowing in > each ceph.conf file, under the general section: > > mon data avail warn = 15 > mon data avail crit = 5 > > I found this settings on ceph mailing list... > > Thanks a lot, > Andrija > > > On 17 June 2014 19:22, Gregory Farnum <g...@inktank.com> wrote: >> >> Try running "ceph health detail" on each of the monitors. Your disk space >> thresholds probably aren't configured correctly or something. >> -Greg >> >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Tue, Jun 17, 2014 at 2:09 AM, Andrija Panic <andrija.pa...@gmail.com> >> wrote: >>> >>> Hi, >>> >>> thanks for that, but is not space issue: >>> >>> OSD drives are only 12% full. >>> and /var drive on which MON lives is over 70% only on CS3 server, but I >>> have increased alert treshold in ceph.conf (mon data avail warn = 15, mon >>> data avail crit = 5), and since I increased them those alerts are gone >>> (anyway, these alerts for /var full over 70% can be normally seen in logs >>> and in ceph -w output). >>> >>> Here I get no normal/visible warning in eather logs or ceph -w output... >>> >>> Thanks, >>> Andrija >>> >>> >>> >>> >>> On 17 June 2014 11:00, Stanislav Yanchev <s.yanc...@maxtelecom.bg> wrote: >>>> >>>> Try grep in cs1 and cs3 could be a disk space issue. >>>> >>>> >>>> >>>> >>>> >>>> Regards, >>>> >>>> Stanislav Yanchev >>>> Core System Administrator >>>> >>>> >>>> >>>> Mobile: +359 882 549 441 >>>> s.yanc...@maxtelecom.bg >>>> www.maxtelecom.bg >>>> >>>> >>>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >>>> Andrija Panic >>>> Sent: Tuesday, June 17, 2014 11:57 AM >>>> To: Christian Balzer >>>> Cc: ceph-users@lists.ceph.com >>>> Subject: Re: [ceph-users] Cluster status reported wrongly as HEALTH_WARN >>>> >>>> >>>> >>>> Hi Christian, >>>> >>>> >>>> >>>> that seems true, thanks. >>>> >>>> >>>> >>>> But again, there are only occurence in GZ logs files (that were >>>> logrotated, not in current log files): >>>> >>>> Example: >>>> >>>> >>>> >>>> [root@cs2 ~]# grep -ir "WRN" /var/log/ceph/ >>>> >>>> Binary file /var/log/ceph/ceph-mon.cs2.log-20140612.gz matches >>>> >>>> Binary file /var/log/ceph/ceph.log-20140614.gz matches >>>> >>>> Binary file /var/log/ceph/ceph.log-20140611.gz matches >>>> >>>> Binary file /var/log/ceph/ceph.log-20140612.gz matches >>>> >>>> Binary file /var/log/ceph/ceph.log-20140613.gz matches >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Andrija >>>> >>>> >>>> >>>> On 17 June 2014 10:48, Christian Balzer <ch...@gol.com> wrote: >>>> >>>> >>>> Hello, >>>> >>>> >>>> On Tue, 17 Jun 2014 10:30:44 +0200 Andrija Panic wrote: >>>> >>>> > Hi, >>>> > >>>> > I have 3 node (2 OSD per node) CEPH cluster, running fine, not much data, >>>> > network also fine: >>>> > Ceph ceph-0.72.2. >>>> > >>>> > When I issue "ceph status" command, I get randomly HEALTH_OK, and >>>> > imidiately after that when repeating command, I get HEALTH_WARN >>>> > >>>> > Examle given down - these commands were issues within less than 1 sec >>>> > between them >>>> > There are NO occuring of word "warn" in the logs (grep -ir "warn" >>>> > /var/log/ceph) on any of the servers... >>>> > I get false alerts with my status monitoring script, for this reason... >>>> > >>>> >>>> If I recall correctly, the logs will show INF, WRN and ERR, so grep for >>>> WRN. >>>> >>>> Regards, >>>> >>>> Christian >>>> >>>> >>>> > Any help would be greatly appriciated. >>>> > >>>> > Thanks, >>>> > >>>> > [root@cs3 ~]# ceph status >>>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>>> > health HEALTH_OK >>>> > monmap e2: 3 mons at >>>> > {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>>> > osdmap e890: 6 osds: 6 up, 6 in >>>> > pgmap v2379904: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>>> > 2576 GB used, 19732 GB / 22309 GB avail >>>> > 448 active+clean >>>> > client io 17331 kB/s rd, 113 kB/s wr, 176 op/s >>>> > >>>> > [root@cs3 ~]# ceph status >>>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>>> > health HEALTH_WARN >>>> > monmap e2: 3 mons at >>>> > {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>>> > osdmap e890: 6 osds: 6 up, 6 in >>>> > pgmap v2379905: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>>> > 2576 GB used, 19732 GB / 22309 GB avail >>>> > 448 active+clean >>>> > client io 28383 kB/s rd, 566 kB/s wr, 321 op/s >>>> > >>>> > [root@cs3 ~]# ceph status >>>> > cluster cab20370-bf6a-4589-8010-8d5fc8682eab >>>> > health HEALTH_OK >>>> > monmap e2: 3 mons at >>>> > {cs1=10.44.xxx.10:6789/0,cs2=10.44.xxx.11:6789/0,cs3=10.44.xxx.12:6789/0}, >>>> > election epoch 122, quorum 0,1,2 cs1,cs2,cs3 >>>> > osdmap e890: 6 osds: 6 up, 6 in >>>> > pgmap v2379913: 448 pgs, 4 pools, 862 GB data, 217 kobjects >>>> > 2576 GB used, 19732 GB / 22309 GB avail >>>> > 448 active+clean >>>> > client io 21632 kB/s rd, 49354 B/s wr, 283 op/s >>>> > >>>> >>>> >>>> -- >>>> >>>> Christian Balzer Network/Systems Engineer >>>> ch...@gol.com Global OnLine Japan/Fusion Communications >>>> http://www.gol.com/ >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> Andrija Panić >>>> >>>> -------------------------------------- >>>> >>>> http://admintweets.com >>>> >>>> -------------------------------------- >>>> >>>> Confidentiality notice >>>> >>>> ________________________________ >>>> >>>> >>>> >>>> The information contained in this message (including any attachments) is >>>> confidential and may be legally privileged or otherwise protected from >>>> disclosure. This message is intended solely for the addressee(s). If you >>>> are not the intended recipient, please notify the sender by return e-mail >>>> and delete this message from your system. Any unauthorised use, >>>> reproduction, or dissemination of this message is strictly prohibited. Any >>>> liability arising from any third party acting, or refraining from acting, >>>> on any information contained in this e-mail is hereby excluded. Please >>>> note that e-mails are susceptible to change. Max Telecom shall not be >>>> liable for the improper or incomplete transmission of the information >>>> contained in this communication, nor shall it be liable for any delay in >>>> its receipt. >>> >>> >>> >>> >>> -- >>> >>> Andrija Panić >>> -------------------------------------- >>> http://admintweets.com >>> -------------------------------------- >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > > > > -- > > Andrija Panić > -------------------------------------- > http://admintweets.com > -------------------------------------- _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
