Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-21 Thread Glyph Lefkowitz
> On Jul 21, 2016, at 5:49 AM, Adi Roiban wrote: > > I can see that when there we got more builds there is significant increase in > memory usage... but will recover once moved to idle. > Cool. Is there something we can do to limit the global concurrency of the builds to preserve resources

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-21 Thread Adi Roiban
On 21 July 2016 at 00:58, Glyph Lefkowitz wrote: > > On Jul 20, 2016, at 2:31 PM, Glyph Lefkowitz > wrote: > > I have recently discovered the Rackspace monitoring capabilities for VM... > and set up a memory notification... not sure who will receive the alerts. > > > I'll make sure that the rele

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-20 Thread Glyph Lefkowitz
> On Jul 20, 2016, at 2:31 PM, Glyph Lefkowitz wrote: > >> I have recently discovered the Rackspace monitoring capabilities for VM... >> and set up a memory notification... not sure who will receive the alerts. > > I'll make sure that the relevant people are on the monitoring list. I created

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-20 Thread Glyph Lefkowitz
> On Jul 20, 2016, at 11:01 AM, Adi Roiban wrote: > > > > On 20 July 2016 at 17:51, Glyph Lefkowitz > wrote: > >> On Jul 20, 2016, at 6:31 AM, Adi Roiban > > wrote: >> >> >> >> On 18 July 2016 at 19:04, James Broadhead >

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-20 Thread Adi Roiban
On 20 July 2016 at 17:51, Glyph Lefkowitz wrote: > > On Jul 20, 2016, at 6:31 AM, Adi Roiban wrote: > > > > On 18 July 2016 at 19:04, James Broadhead > wrote: > >> On 17 July 2016 at 07:21, Amber Brown wrote: >> >>> It's OOMing (...) >>> >> >> Have you considered something like monit[1] to de

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-20 Thread Glyph Lefkowitz
> On Jul 20, 2016, at 6:31 AM, Adi Roiban wrote: > > > > On 18 July 2016 at 19:04, James Broadhead > wrote: > On 17 July 2016 at 07:21, Amber Brown > wrote: > It's OOMing (...) > > > Have you considered something like moni

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-20 Thread Adi Roiban
On 18 July 2016 at 19:04, James Broadhead wrote: > On 17 July 2016 at 07:21, Amber Brown wrote: > >> It's OOMing (...) >> > > Have you considered something like monit[1] to detect & restart in cases > like this? > > This might help, but will not help up understand what we are doing wrong :) Af

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-18 Thread James Broadhead
On 17 July 2016 at 07:21, Amber Brown wrote: > It's OOMing (...) > Have you considered something like monit[1] to detect & restart in cases like this? [1] https://mmonit.com/monit/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-16 Thread Adi Roiban
On 17 July 2016 at 07:38, Amber Brown wrote: > Yeah, that's a good idea - disable them for now, and we'll see if the OOMs > happen. Then we can investigate them closer if it stops. > > On 17 Jul 2016 08:37, "Adi Roiban" wrote: > >> >> >> On 17 July 2016 at 07:21, Amber Brown wrote: >> >>> It's

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-16 Thread Amber Brown
Yeah, that's a good idea - disable them for now, and we'll see if the OOMs happen. Then we can investigate them closer if it stops. On 17 Jul 2016 08:37, "Adi Roiban" wrote: > > > On 17 July 2016 at 07:21, Amber Brown wrote: > >> It's OOMing -- I think the upgrade to Eight trunk introduced some

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-16 Thread Adi Roiban
On 17 July 2016 at 07:21, Amber Brown wrote: > It's OOMing -- I think the upgrade to Eight trunk introduced some sort of > memory usage regression or we've done something wrong -- I've unfortunately > not had time to investigate. > > We could size up the RAM in the meantime I guess? > > -Amber >

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-16 Thread Amber Brown
It's OOMing -- I think the upgrade to Eight trunk introduced some sort of memory usage regression or we've done something wrong -- I've unfortunately not had time to investigate. We could size up the RAM in the meantime I guess? -Amber On 17 Jul 2016 08:19, "Adi Roiban" wrote: > > > On 17 July

Re: [Twisted-Python] buildbot.twistedmatrix.com is down a lot

2016-07-16 Thread Adi Roiban
On 17 July 2016 at 06:11, Craig Rodrigues wrote: > In the past few days, buildbot.twistedmatrix.com seems to be down all the > time, and requires manual restarts. As I write this, it is down right now. > > Is there something wrong with the hardware involved with > buildbot.twistedmatrix.com? > >