Gordon,
The Qpid Broker memory started spiking again, its usage is 50% now, is there anything specific I should be looking at in the logs, can you elucidate some pointers to look at to figure the root cause. ________________________________ From: Venkat Jaligama Sent: Friday, March 1, 2019 11:09:26 AM To: Chester Knapp; In Ho Sok; users@qpid.apache.org Subject: Re: Issue with Qpid Memory Gordon, Previously attached logs consists of Log metadata also, the actual log messages are as follows 2019-02-28 13:07:44 [System] error [0x7f89e3fff700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:44 [System] error [0x7f89e9d18700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:44 [System] error [0x7f89e8d16700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43 [System] error [0x7f89e2ffd700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43 [System] error [0x7f89e2ffd700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43 [System] error [0x7f89ead1a700] Error reading socket: Encountered end of file [-5938] Does these error indicate something with the memory issue, out of 3.4 million logs 2.4 million logs are these error messages which is almost 70 percent ________________________________ From: Venkat Jaligama Sent: Thursday, February 28, 2019 6:06:02 PM To: Chester Knapp; In Ho Sok; users@qpid.apache.org Subject: Re: Issue with Qpid Memory Gordon, I have see the Logs of the Qpid container which had high memory usage, application logs seem to be normal however I see many error logs. These error logs constitute to 70% of the total logs generated by Qpid Container. Below are some of the error logs. Could these error logs indicate something? =============================== 2019-02-28 13:07:44.171 ip-10-251-5-215 2019-02-28 13:07:44 [System] error [0x7f89e3fff700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:44.171 ip-10-251-5-215 2019-02-28 13:07:44 [System] error [0x7f89e9d18700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:44.171 ip-10-251-5-215 2019-02-28 13:07:44 [System] error [0x7f89e8d16700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43.468 ip-10-251-5-215 2019-02-28 13:07:43 [System] error [0x7f89e2ffd700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43.467 ip-10-251-5-215 2019-02-28 13:07:43 [System] error [0x7f89e2ffd700] Error reading socket: Encountered end of file [-5938] 2019-02-28 13:07:43.464 ip-10-251-5-215 2019-02-28 13:07:43 [System] error [0x7f89ead1a700] Error reading socket: Encountered end of file [-5938] ================================= ________________________________ From: Chester Knapp Sent: Thursday, February 28, 2019 10:29:01 AM To: Venkat Jaligama; In Ho Sok Subject: Fw: Issue with Qpid Memory ________________________________ From: Chester <knap...@gmail.com> Sent: Thursday, February 28, 2019 10:26 AM To: Chester Knapp Subject: Fwd: Issue with Qpid Memory ---------- Forwarded message --------- From: Gordon Sim <g...@redhat.com<mailto:g...@redhat.com>> Date: Thu, Feb 28, 2019 at 9:25 AM Subject: Re: Issue with Qpid Memory To: <users@qpid.apache.org<mailto:users@qpid.apache.org>> On 28/02/2019 1:58 pm, Chester wrote: > Gordon, > > We see several days of low/normal (.1 GB) memory usage, followed by a > linear increase of about .15 GB per hour, until we OOM after about 24 > hours. I've attached some graphs and a spreadsheet to help visualize. > 1week_memory_usage.png > 30day_memory_usage.png > > We have not identified any pattern of activity associated with the > changing memory utilization. Our workload has a fairly constant > enqueue/dequeue rate. I think trying to find what else changes when the memory starts growing would be the best way to progress. I assume there is nothing in the logs? Perhaps try getting qpid-stat queries periodically to try and see if the differences over time show anything? --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org<mailto:users-unsubscr...@qpid.apache.org> For additional commands, e-mail: users-h...@qpid.apache.org<mailto:users-h...@qpid.apache.org>
