i'll take a look at this today.
On Mon, Sep 8, 2014 at 1:13 AM, Josh Rosen <rosenvi...@gmail.com> wrote: > Yeah, I think https://github.com/apache/spark/pull/2315 should have fixed > the Mima issue. We're still seeing some intermittent failures due to > DriverSuite and SparkSubmitSuite tests failing, so I'd appreciate any help > in diagnosing that issue. > > On Sun, Sep 7, 2014 at 10:08 PM, Prashant Sharma <scrapco...@gmail.com> > wrote: > >> Looks like this is already taken care of ? >> >> Prashant Sharma >> >> >> >> On Mon, Sep 8, 2014 at 4:37 AM, Josh Rosen <rosenvi...@gmail.com> wrote: >> >>> Does anyone know why some of the MiMa tests have started failing? >>> >>> See >>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19948/consoleFull >>> for >>> an example. >>> >>> On September 6, 2014 at 12:48:27 PM, Josh Rosen (rosenvi...@gmail.com) >>> wrote: >>> >>> It looks like Jenkins is up and running, but there seems to be a delay >>> in responding to requests to re-test patches. It seems like Jenkins is >>> promptly testing new PRs, or new commits as they’re added to existing PRs, >>> but taking a very long time to respond to requests to re-test PRs. >>> >>> I’m going to continue monitoring this today. I’m considering creating >>> my own fork of the Jenkins pull request builder plugin so that we can add >>> extra logging in order to diagnose what’s causing this lag. >>> >>> - Josh >>> On September 5, 2014 at 5:49:32 PM, Josh Rosen (rosenvi...@gmail.com) >>> wrote: >>> >>> We have successfully purged Jenkins’ build queue. If you want a PR to >>> be re-tested, please ask Jenkins again. >>> >>> On September 5, 2014 at 5:36:30 PM, shane knapp (skn...@berkeley.edu) >>> wrote: >>> >>> yeah, it was a problem w/the PRB's OAuth key. josh rosen added a new key, >>> and magique! >>> >>> we're about to clear the queue of all builds as most aren't >>> wanted/needed. >>> >>> >>> On Fri, Sep 5, 2014 at 5:33 PM, Nicholas Chammas < >>> nicholas.cham...@gmail.com >>> > wrote: >>> >>> > Looks like Jenkins is back! >>> > >>> > lol The poor guy has like a million builds >>> > < >>> https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/ >>> > >>> > to catch up on. >>> > >>> > >>> > On Fri, Sep 5, 2014 at 4:15 PM, Nicholas Chammas < >>> > nicholas.cham...@gmail.com> wrote: >>> > >>> >> How's it going? >>> >> >>> >> It looks like during the last build >>> >> < >>> https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/lastBuild/console >>> > >>> >> from about 30 min ago Jenkins was still having trouble fetching from >>> >> GitHub. It also looks like not all requests for testing are triggering >>> >> builds. >>> >> >>> >> >>> >> On Fri, Sep 5, 2014 at 1:23 PM, shane knapp <skn...@berkeley.edu> >>> wrote: >>> >> >>> >>> it's looking like everything except the pull request builders are >>> >>> working. i'm going to be working on getting this resolved today. >>> >>> >>> >>> >>> >>> On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas < >>> >>> nicholas.cham...@gmail.com> wrote: >>> >>> >>> >>>> Hmm, looks like at least some builds >>> >>>> < >>> https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull >>> > >>> >>>> are working now, though this last one was from ~5 hours ago. >>> >>>> >>> >>>> >>> >>>> On Fri, Sep 5, 2014 at 1:02 AM, shane knapp <skn...@berkeley.edu> >>> >>>> wrote: >>> >>>> >>> >>>>> yep. that's exactly the behavior i saw earlier, and will be >>> figuring >>> >>>>> out first thing tomorrow morning. i bet it's an environment issues >>> on the >>> >>>>> slaves. >>> >>>>> >>> >>>>> >>> >>>>> On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas < >>> >>>>> nicholas.cham...@gmail.com> wrote: >>> >>>>> >>> >>>>>> Looks like during the last build >>> >>>>>> < >>> https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console >>> > >>> >>>>>> Jenkins was unable to execute a git fetch? >>> >>>>>> >>> >>>>>> >>> >>>>>> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp <skn...@berkeley.edu> >>> >>>>>> wrote: >>> >>>>>> >>> >>>>>>> i'm going to restart jenkins and see if that fixes things. >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp <skn...@berkeley.edu >>> > >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>>> looking >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < >>> >>>>>>>> nicholas.cham...@gmail.com> wrote: >>> >>>>>>>> >>> >>>>>>>>> It appears that our main man is having trouble >>> >>>>>>>>> < >>> https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/ >>> > >>> >>>>>>>>> hearing new requests >>> >>>>>>>>> < >>> https://github.com/apache/spark/pull/2277#issuecomment-54549106>. >>> >>>>>>>>> >>> >>>>>>>>> Do we need some smelling salts? >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp < >>> skn...@berkeley.edu> >>> >>>>>>>>> wrote: >>> >>>>>>>>> >>> >>>>>>>>>> i'd ping the Jenkinsmench... the master was completely >>> offline, >>> >>>>>>>>>> so any new >>> >>>>>>>>>> jobs wouldn't have reached it. any jobs that were queued when >>> >>>>>>>>>> power was >>> >>>>>>>>>> lost probably started up, but jobs that were running would >>> fail. >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas < >>> >>>>>>>>>> nicholas.cham...@gmail.com >>> >>>>>>>>>> > wrote: >>> >>>>>>>>>> >>> >>>>>>>>>> > Woohoo! Thanks Shane. >>> >>>>>>>>>> > >>> >>>>>>>>>> > Do you know if queued PR builds will automatically be picked >>> >>>>>>>>>> up? Or do we >>> >>>>>>>>>> > have to ping the Jenkinmensch manually from each PR? >>> >>>>>>>>>> > >>> >>>>>>>>>> > Nick >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp < >>> >>>>>>>>>> skn...@berkeley.edu> wrote: >>> >>>>>>>>>> > >>> >>>>>>>>>> >> AND WE'RE UP! >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> sorry that this took so long... i'll send out a more >>> detailed >>> >>>>>>>>>> explanation >>> >>>>>>>>>> >> of what happened soon. >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> now, off to back up jenkins. >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> shane >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp < >>> >>>>>>>>>> skn...@berkeley.edu> wrote: >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> > it's a faulty power switch on the firewall, which has >>> been >>> >>>>>>>>>> swapped out. >>> >>>>>>>>>> >> > we're about to reboot and be good to go. >>> >>>>>>>>>> >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp < >>> >>>>>>>>>> skn...@berkeley.edu> >>> >>>>>>>>>> >> wrote: >>> >>>>>>>>>> >> > >>> >>>>>>>>>> >> >> looks like some hardware failed, and we're swapping in a >>> >>>>>>>>>> replacement. >>> >>>>>>>>>> >> i >>> >>>>>>>>>> >> >> don't have more specific information yet -- including >>> >>>>>>>>>> *what* failed, >>> >>>>>>>>>> >> as our >>> >>>>>>>>>> >> >> sysadmin is super busy ATM. the root cause was an >>> >>>>>>>>>> incorrect circuit >>> >>>>>>>>>> >> being >>> >>>>>>>>>> >> >> switched off during building maintenance. >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> >> on a side note, this incident will be accelerating our >>> plan >>> >>>>>>>>>> to move the >>> >>>>>>>>>> >> >> entire jenkins infrastructure in to a managed datacenter >>> >>>>>>>>>> environment. >>> >>>>>>>>>> >> this >>> >>>>>>>>>> >> >> will be our major push over the next couple of weeks. >>> more >>> >>>>>>>>>> details >>> >>>>>>>>>> >> about >>> >>>>>>>>>> >> >> this, also, as soon as i get them. >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> >> i'm very sorry about the downtime, we'll get everything >>> up >>> >>>>>>>>>> and running >>> >>>>>>>>>> >> >> ASAP. >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp < >>> >>>>>>>>>> skn...@berkeley.edu> >>> >>>>>>>>>> >> wrote: >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> >>> looks like a power outage in soda hall. more updates as >>> >>>>>>>>>> they happen. >>> >>>>>>>>>> >> >>> >>> >>>>>>>>>> >> >>> >>> >>>>>>>>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp < >>> >>>>>>>>>> skn...@berkeley.edu> >>> >>>>>>>>>> >> >>> wrote: >>> >>>>>>>>>> >> >>> >>> >>>>>>>>>> >> >>>> i am trying to get things up and running, but it looks >>> >>>>>>>>>> like either >>> >>>>>>>>>> >> the >>> >>>>>>>>>> >> >>>> firewall gateway or jenkins server itself is down. >>> i'll >>> >>>>>>>>>> update as >>> >>>>>>>>>> >> soon as >>> >>>>>>>>>> >> >>>> i know more. >>> >>>>>>>>>> >> >>>> >>> >>>>>>>>>> >> >>> >>> >>>>>>>>>> >> >>> >>> >>>>>>>>>> >> >> >>> >>>>>>>>>> >> > >>> >>>>>>>>>> >> >>> >>>>>>>>>> > >>> >>>>>>>>>> > -- >>> >>>>>>>>>> > You received this message because you are subscribed to the >>> >>>>>>>>>> Google Groups >>> >>>>>>>>>> > "amp-infra" group. >>> >>>>>>>>>> > To unsubscribe from this group and stop receiving emails >>> from >>> >>>>>>>>>> it, send an >>> >>>>>>>>>> > email to amp-infra+unsubscr...@googlegroups.com. >>> >>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>> >>>>>>>>>> > >>> >>>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>>>>> >>> >>>>>> >>> >>>>> >>> >>>> >>> >>> >>> >> >>> > >>> >> >> >
