Ugo, Please do proper log analysis, otherwise we can only guess.
If you restart network with clean up - that means a BRAND NEW VR was created for your networks.. That sounds like something was wrong with old one. And that needs investigation of log (mgmt and agent) and/or inside VR, before you destroy it. I don't get the DNS part of your reply... Kind regards, Andrija [email protected] www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -----Original Message----- From: Ugo Vasi <[email protected]> Sent: 07 February 2019 11:47 To: [email protected]; Andrija Panic <[email protected]> Subject: Re: Mysql got timeout and problems with UI Hi Adrija, I restarted the network with the appropriate button (enabling the clean option) and then the firewall responds correctly, this makes me think that it is not a problem of communication between browsers and mgmt. It would seem to me a DNS problem that delays the answers, how can I check the calls that the software agent makes inside the router? Is there a query to count the number of rules of that firewall or for ACL? I've never used more than 15 rules for single IP. Il 07/02/19 09:19, Andrija Panic ha scritto: > Hi Ugo, > > I have never observed such behavior in any release so far (general > sluggishness of GUI) - I hope you don't mind me asking you to make sure that > your network connectivity is good enough and so on, between browser/API > client and the mgmt server itself (and of course between mgmt and mysql > server) - reason being that in some of my ex companies I got one (high level) > guy complaining on lost packet when doing ping to the DC, but it was his WiFi > which has problems (once moved to wire, everything magically worked). > > To be fair, I did see dead gui when a customer has been abusing the ACL > rules, i.e. had like 200 rules on single ACL, and browser could not handle > showing it and it was completely unmanageable. > > As for other issue, this seems to be something you had before (if I remember > ML correctly) - please check the logs inside VR and any lines in mgmt. log > ABOVE that final error string (usually it will tell you that i.e. ssh > connection to VR could not be done). > > Also worth checking Agent logs on specific host where VR is running (make > sure to increase log level to debug first) > For MySQL issues, again check network, and make sure no LB with short TCP > timeout is set in between. > > Kind regards, > Andrija > > [email protected] > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > > > -----Original Message----- > From: Ugo Vasi <[email protected]> > Sent: 06 February 2019 14:53 > To: [email protected] > Subject: Re: Mysql got timeout and problems with UI > > Hi all, > adding a new firewall rule I receive this error (note, after 1189 seconds): > > 2019-02-06 14:42:08,441 WARN [o.a.c.f.j.i.AsyncJobMonitor] > (Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for > 1189 seconds > 2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request] > (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq > 7-4193977152988777787: Received: { Ans: , MgmtId: 220777304233416, > via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } } > 2019-02-06 14:42:20,358 WARN [c.c.n.f.FirewallManagerImpl] > (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) > Failed to apply firewall rules due to : Resource [DataCenter:1] is > unreachable: Unable to apply firewall rules on router > 2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async > job-886, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > to create firewall rule"} > > > The rule now is in "Deleting" state. > > > Il 06/02/19 10:59, Ugo Vasi ha scritto: >> Hi all, >> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced >> network) I have some problems in cloudstack usage. >> >> During the usual operations, I find a slowdown in the responses of the >> UI and occasionally errors. >> >> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of >> a specific network (net-name). >> >> The UI has stopped in the assignment phase even if in the logs (which >> I report at the bottom of the email) I see that the job is executed in >> a few seconds and updating the list of associated IPs in the network >> list I see that the association is correct. >> >> I noticed that a series of these messages appear in the mysql log: >> >> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db: >> 'dbname' user: 'username' host: 'localhost' (Got timeout reading >> communication packets) 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted >> connection 6269 to db: >> 'dbname' user: 'username' host: 'localhost' (Got timeout reading >> communication packets) 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted >> connection 6289 to db: >> 'dbname' user: 'username' host: 'localhost' (Got timeout reading >> communication packets) 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted >> connection 6101 to db: >> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading >> communication packets >> >> >> Note that the mysql server is used only by cloustack and is configured >> according to the instructions in the cloudstack manual. >> >> I've already had a number of similar problems before (the version of >> ACS was 4.11.1) and the situation has been getting worse and worse >> until the interface and the whole cloudstack system has stopped >> working (see "URGENT: Unable to schedule async job for command >> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to >> apply firewall rules on router"). >> >> >> Has anyone had similar experiences or can help me understand where the >> problem is? >> >> >> >> Job log (extracted from management-server.log): >> >> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876 >> into job monitoring 2019-02-06 10:21:07,294 DEBUG >> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a >> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2, >> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd: >> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd >> min, >> cmdInfo: >> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0 >> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31", >> "ctxDetails":"{\"interface >> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\" >> ,\"interface >> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}" >> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd >> EventType":"NET.IPASSIGN","_":"1549444854378"}, >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >> result: null, initMsid: 220777304233416, completeMsid: null, >> lastUpdated: null, lastPolled: null, created: null} 2019-02-06 >> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit >> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2, >> instanceType: IpAddress, instanceId: 31, cmd: >> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd >> min, >> cmdInfo: >> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0 >> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31", >> "ctxDetails":"{\"interface >> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\" >> ,\"interface >> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}" >> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd >> EventType":"NET.IPASSIGN","_":"1549444854378"}, >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >> result: null, initMsid: 220777304233416, completeMsid: null, >> lastUpdated: null, lastPolled: null, created: null} 2019-02-06 >> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network >> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG >> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a >> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type = >> public_ip count for Account = 2 Operation = increasing Amount = 1 >> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to >> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG >> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a >> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876, >> jobStatus: SUCCEEDED, resultCode: 0, result: >> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":" >> a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","alloc >> ated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88 >> d7d429107","zonename":"somezone","issourcenat":false,"account":"admin" >> ,"domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","fo >> rvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4", >> "vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"ass >> ociatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatedne >> tworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d >> 6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-7 >> 8d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true} >> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Publish async job-876 complete on message bus >> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06 >> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442 >> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) >> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all >> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG >> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a >> job-876) (logid:ce8dc8fb) Done executing >> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd >> min for job-876 2019-02-06 10:21:07,447 INFO >> [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-6:ctx-5d20ed8a >> job-876) (logid:ce8dc8fb) Remove >> job-876 from job monitoring >> >> > -- *Ugo Vasi* / System Administrator [email protected] <mailto:[email protected]> *Procne S.r.l.* +39 0432 486 523 via Cotonificio, 45 33010 Tavagnacco (UD) www.procne.it <http://www.procne.it/> Le informazioni contenute nella presente comunicazione ed i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone od alla Società sopraindicati. La diffusione, distribuzione e/o copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei dati personali". Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail [email protected] <mailto:[email protected]>.
