Re: [Openstack] BUG: soft lockup messages

2015-02-03 Thread Matej Mailing
Hi Giacomo, sorry for a late reply. It turned out that the major improvement was made through increasing NFS packet size from 8k to 32k. After that no more of those messages were shown and everything is working just great :-) Thank you everyone for your suggestions and ideas again. Matej 201

Re: [Openstack] BUG: soft lockup messages

2015-01-13 Thread Giacomo Tenaglia
On 09/01/15 13:01, Matej Mailing wrote: we have tweaked some NFS parameters for our backend storage, including packet sizes and now the load is constantly slower, there is much less I/O waiting. There were no lock-ups after that. So it really seems it was slow storage that was to be blamed. I wil

Re: [Openstack] BUG: soft lockup messages

2015-01-09 Thread Matej Mailing
2015-01-07 13:07 GMT+01:00 Ritesh Raj Sarraf : > On 01/07/2015 04:32 PM, Tim Bell wrote: > >> The profile of this hardware looks virtualized. That makes me ask if you >> have >> installed the para-virtualized drivers ? >> > > Out of interest, does using para-virtualized drivers reduce the chance of

Re: [Openstack] BUG: soft lockup messages

2015-01-07 Thread Ritesh Raj Sarraf
On 01/07/2015 04:32 PM, Tim Bell wrote: >> > The profile of this hardware looks virtualized. That makes me ask if you >> > have >> > installed the para-virtualized drivers ? >> > > Out of interest, does using para-virtualized drivers reduce the chance of > soft lockups ? I can understand that it

Re: [Openstack] BUG: soft lockup messages

2015-01-07 Thread Matej Mailing
2015-01-07 9:06 GMT+01:00 Ritesh Raj Sarraf : > > > On 01/05/2015 03:46 PM, Matej Mailing wrote: >> Hello Erhan, >> >> soft lock-up has just happened again on the instance node. I am >> monitoring network interface traffic on which the NFS server is >> connected and the interface has constantly bee

Re: [Openstack] BUG: soft lockup messages

2015-01-07 Thread Tim Bell
> -Original Message- > From: Ritesh Raj Sarraf [mailto:r...@researchut.com] > Sent: 07 January 2015 09:06 > To: openstack@lists.openstack.org > Subject: Re: [Openstack] BUG: soft lockup messages > > > The profile of this hardware looks virtualized. That

Re: [Openstack] BUG: soft lockup messages

2015-01-07 Thread Matej Mailing
2015-01-07 9:06 GMT+01:00 Ritesh Raj Sarraf : > > > On 01/05/2015 03:46 PM, Matej Mailing wrote: >> Hello Erhan, >> >> soft lock-up has just happened again on the instance node. I am >> monitoring network interface traffic on which the NFS server is >> connected and the interface has constantly bee

Re: [Openstack] BUG: soft lockup messages

2015-01-07 Thread Ritesh Raj Sarraf
On 01/05/2015 03:46 PM, Matej Mailing wrote: > Hello Erhan, > > soft lock-up has just happened again on the instance node. I am > monitoring network interface traffic on which the NFS server is > connected and the interface has constantly been under 20% of it's > capacity, also the load on the N

Re: [Openstack] BUG: soft lockup messages

2015-01-05 Thread Matej Mailing
2015-01-04 21:20 GMT+01:00 Erhan Ekici : > Hi Matej, > > Can you change the values(both on compute and instance node) and see if soft > lock-ups generated again? There are two approaches; one decreasing and the > other increasing the cache. > > vm.dirty_background_ratio = 5 > vm.dirty_ratio = 10 >

Re: [Openstack] BUG: soft lockup messages

2015-01-05 Thread Matej Mailing
2015-01-04 21:20 GMT+01:00 Erhan Ekici : > Hi Matej, > > Can you change the values(both on compute and instance node) and see if soft > lock-ups generated again? There are two approaches; one decreasing and the > other increasing the cache. > > vm.dirty_background_ratio = 5 > vm.dirty_ratio = 10 >

Re: [Openstack] BUG: soft lockup messages

2015-01-04 Thread Erhan Ekici
Hi Matej, Can you change the values(both on compute and instance node) and see if soft lock-ups generated again? There are two approaches; one decreasing and the other increasing the cache. vm.dirty_background_ratio = 5 vm.dirty_ratio = 10 If I/O & Cache is not the source of the problem, we shou

Re: [Openstack] BUG: soft lockup messages

2015-01-04 Thread Matej Mailing
2015-01-04 17:10 GMT+01:00 Erhan Ekici : > Hi Matej, > > I am not sure but most of the soft lock-up messages generated due to I/O > issues. From your log entries, I see mysqld issues as well as others: > > Jan 3 10:41:44 postar kernel: [303154.352793] NMI watchdog: BUG: soft > lockup - CPU#3 stuc

Re: [Openstack] BUG: soft lockup messages

2015-01-04 Thread Erhan Ekici
Hi Matej, I am not sure but most of the soft lock-up messages generated due to I/O issues. From your log entries, I see mysqld issues as well as others: Jan 3 10:41:44 postar kernel: [303154.352793] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 45s! [mysqld:4601] Jan 3 10:44:00 postar

Re: [Openstack] BUG: soft lockup messages

2015-01-04 Thread Matej Mailing
2015-01-03 17:56 GMT+01:00 John Griffith : > On Sat, Jan 3, 2015 at 5:05 AM, Matej Mailing wrote: >> Hi, >> >> we are experiencing randomly-timed soft lockup messages in different >> instances (running CentOS, Ubuntu, etc.) with different processes and >> on different compute nodes. I suspect that

Re: [Openstack] BUG: soft lockup messages

2015-01-03 Thread John Griffith
On Sat, Jan 3, 2015 at 5:05 AM, Matej Mailing wrote: > Hi, > > we are experiencing randomly-timed soft lockup messages in different > instances (running CentOS, Ubuntu, etc.) with different processes and > on different compute nodes. I suspect that access to the cinder > storage via NFS could be t

[Openstack] BUG: soft lockup messages

2015-01-03 Thread Matej Mailing
Hi, we are experiencing randomly-timed soft lockup messages in different instances (running CentOS, Ubuntu, etc.) with different processes and on different compute nodes. I suspect that access to the cinder storage via NFS could be the cause of issue, but then perhaps all the instances would trigg