Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread John Baldwin
On Wednesday 13 May 2009 1:44:55 pm Marc G. Fournier wrote: > On Wed, 13 May 2009, John Baldwin wrote: > > > Well, you had a whole lot of page faults and other VM activity, plus 500k > > syscalls. The 'w' is a count of swapped processes, so basically your box is > > swapping a whole lot it seems.

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Steven Hartland
- Original Message - From: "Marc G. Fournier" We'll see hwo the next 'test period' works out, with that MySQL stuff offline ... the other thing I've been working on is moving jails off of that server, one at a time, to see if I can narrow down which one is causing the spike ... I will

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
On Wed, 13 May 2009, Steven Hartland wrote: We've seen things similar to this when an process uncommon process does a query which locks the a table for a large amount of time on mysql. So many reasons why I hate MySQL :( One thing that we are trying right now is actually along these lines

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Steven Hartland
- Original Message - From: "Marc G. Fournier" Right now, IO is running ~775 processes ... at the time of the vmstat I provided earlier, it was up to 1400 processes ... since there is only 5 minutes between script runs, something is causing it to go from zero swap -> high swap within

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
On Wed, 13 May 2009, John Baldwin wrote: Well, you had a whole lot of page faults and other VM activity, plus 500k syscalls. The 'w' is a count of swapped processes, so basically your box is swapping a whole lot it seems. I think your box is just overloaded. I knew I was going to regret post

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Chuck Swiger
Hi-- On May 13, 2009, at 9:52 AM, John Baldwin wrote: [ ... ] Well, you had a whole lot of page faults and other VM activity, plus 500k syscalls. The 'w' is a count of swapped processes, so basically your box is swapping a whole lot it seems. I think your box is just overloaded. Yep. Th

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Mike Tancsa
At 12:31 PM 5/13/2009, Marc G. Fournier wrote: On Wed, 13 May 2009, Mike Tancsa wrote: What does your kernel config look like ? Included below ... only thought I had, taht I haven't tried yet, was changing from SCHED_4BSD -> SCHED_ULE ... ULE for sure. Are you sure some of the options be

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread John Baldwin
On Wednesday 13 May 2009 12:34:39 pm Marc G. Fournier wrote: > On Wed, 13 May 2009, John Baldwin wrote: > > > On Wednesday 13 May 2009 3:09:33 am Marc G. Fournier wrote: > >> > >> Don't know if this helps with anything, but it just hung after 2days again > >> ... nothing on the console ... top pr

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
On Wed, 13 May 2009, John Baldwin wrote: On Wednesday 13 May 2009 3:09:33 am Marc G. Fournier wrote: Don't know if this helps with anything, but it just hung after 2days again ... nothing on the console ... top process running at the time shows the following ... anything there look "concerning

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
On Wed, 13 May 2009, Mike Tancsa wrote: What does your kernel config look like ? Included below ... only thought I had, taht I haven't tried yet, was changing from SCHED_4BSD -> SCHED_ULE ... machine amd64 cpu HAMMER ident kernel options SMP option

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Mike Tancsa
At 10:50 AM 5/13/2009, Marc G. Fournier wrote: On Wed, 13 May 2009, John Baldwin wrote: On Wednesday 13 May 2009 3:09:33 am Marc G. Fournier wrote: Don't know if this helps with anything, but it just hung after 2days again ... nothing on the console ... top process running at the time shows t

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
On Wed, 13 May 2009, John Baldwin wrote: On Wednesday 13 May 2009 3:09:33 am Marc G. Fournier wrote: Don't know if this helps with anything, but it just hung after 2days again ... nothing on the console ... top process running at the time shows the following ... anything there look "concerning

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread John Baldwin
On Wednesday 13 May 2009 3:09:33 am Marc G. Fournier wrote: > > Don't know if this helps with anything, but it just hung after 2days again > ... nothing on the console ... top process running at the time shows the > following ... anything there look "concerning"? Is this a 2 CPU system? If so,

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread NAKAJI Hiroyuki
Marc, and folks, I have simillar "hang" problem on 6.4-STABLE and 7.2-STABLE servers, on which apache, squid, inn, named, isc-dhcpd and so on are running except DB servers. What kind of informations should I check to solve this annoying problem? I'm running munin-node on these machines, too. Th

Re: More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Xin LI
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Marc G. Fournier wrote: > > Don't know if this helps with anything, but it just hung after 2days > again ... nothing on the console ... top process running at the time > shows the following ... anything there look "concerning"? Looks like a dead/live

More data on 7.2-RELEASE "hangs"

2009-05-13 Thread Marc G. Fournier
Don't know if this helps with anything, but it just hung after 2days again ... nothing on the console ... top process running at the time shows the following ... anything there look "concerning"? last pid: 5196; load averages: 9.25, 15.97, 10.07 up 2+07:58:36 04:02:28 1874 processes:317