Hi, any hint on how to proceed ?
on haproxy I see rougly 50%/50% sessions across 2 backend servers. But inside DB, it all points to the one mgmt_server_ip... Thanks, Andrija On 4 June 2015 at 19:27, Andrija Panic <andrija.pa...@gmail.com> wrote: > And if of any help another hint: > > while Im having this lines sent to logs in high volume...if I stop second > mgmt server, first one (that is making all these lines, doesnt stop to make > them), so log is still heavily writen to - only when I also restart mgmt on > 1st node (2nd node is down), then these log lines dissapear. > > Thx > > On 4 June 2015 at 19:19, Andrija Panic <andrija.pa...@gmail.com> wrote: > >> And I could add - these lines (in this volume) only appears on first mgmt >> server (Actually I have 2 separate, but identical ACS installations, and >> same behaviour). >> >> On 4 June 2015 at 19:18, Andrija Panic <andrija.pa...@gmail.com> wrote: >> >>> Just checked, in the HOSTS table, all agents are connected (via haproxy) >>> to the first mgmt server...I just restarted haproxy, and still inside the >>> DB, it says same mgmt_server_id for all agents - which is not really true. >>> >>> Actually, on the haproxy itslef (statistics page) I can see almoust >>> 50%-50% distribution across 2 backends - which means by haproxy it should >>> be fine. >>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS mgmt >>> server) >>> >>> This is our haproxy config, I think it's fine, but... DB says >>> differently, althouh haproxy statistick say all fine >>> >>> ### ACS 8250 >>> ####################################################################################### >>> frontend front_ACS_8250 10.20.10.100:8250 >>> option tcplog >>> mode tcp >>> default_backend back_8250 >>> backend back_8250 >>> mode tcp >>> balance source >>> server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000 >>> rise 3 fall 3 >>> server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000 >>> rise 3 fall 3 >>> >>> ################################################################################################## >>> >>> Any info on how to proceed with this, since because of these lines, it >>> makes mgmt logs almoust unreadable... :( >>> >>> Thanks, >>> Andrija >>> >>> On 4 June 2015 at 19:00, Andrija Panic <andrija.pa...@gmail.com> wrote: >>> >>>> Thanks Koushik, >>>> >>>> I will check and let you know - but 11GB log file for 10h ? I dont >>>> expect this is expected :) >>>> I understand that the message is there because of setup, just an awful >>>> lot of lines.... >>>> >>>> Will check thx for the help ! >>>> >>>> Andrija >>>> >>>> On 4 June 2015 at 18:53, Koushik Das <koushik....@citrix.com> wrote: >>>> >>>>> This is expected in a clustered MS setup. What is the distribution of >>>>> HV hosts across these MS (check host table in db for MS id)? MS owning the >>>>> HV host processes all commands for that host. >>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in >>>>> both MS logs to correlate. >>>>> >>>>> >>>>> >>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <andrija.pa...@gmail.com> >>>>> wrote: >>>>> >>>>> > Hi, >>>>> > >>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and >>>>> sometimes it >>>>> > happens that on the first node, we have extremem number of folowing >>>>> line >>>>> > entries in the log fie, which causes many GB log in just few hours >>>>> or less: >>>>> > (as you can see here they are not even that frequent, but sometimes, >>>>> it >>>>> > gets really crazy with the speed/numer logged per seconds: >>>>> > >>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId >>>>> > 90520745449919: Resp: Routing to peer >>>>> > >>>>> > >>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack >>>>> agents >>>>> > (agent.properties file). >>>>> > >>>>> > Any suggestions, how to avoid this - I noticed when I turn off >>>>> second ACS >>>>> > MGMT server, and then reboot first one (restart >>>>> cloudstack-management) it >>>>> > stops and behaves nice :) >>>>> > >>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes. >>>>> > >>>>> > Thanks, >>>>> > -- >>>>> > >>>>> > Andrija Panić >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Andrija Panić >>>> >>> >>> >>> >>> -- >>> >>> Andrija Panić >>> >> >> >> >> -- >> >> Andrija Panić >> > > > > -- > > Andrija Panić > -- Andrija Panić
