yeah, it was a problem w/the PRB's OAuth key. josh rosen added a new key, and magique!
we're about to clear the queue of all builds as most aren't wanted/needed. On Fri, Sep 5, 2014 at 5:33 PM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > Looks like Jenkins is back! > > lol The poor guy has like a million builds > <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> > to catch up on. > > > On Fri, Sep 5, 2014 at 4:15 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> How's it going? >> >> It looks like during the last build >> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/lastBuild/console> >> from about 30 min ago Jenkins was still having trouble fetching from >> GitHub. It also looks like not all requests for testing are triggering >> builds. >> >> >> On Fri, Sep 5, 2014 at 1:23 PM, shane knapp <skn...@berkeley.edu> wrote: >> >>> it's looking like everything except the pull request builders are >>> working. i'm going to be working on getting this resolved today. >>> >>> >>> On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas < >>> nicholas.cham...@gmail.com> wrote: >>> >>>> Hmm, looks like at least some builds >>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull> >>>> are working now, though this last one was from ~5 hours ago. >>>> >>>> >>>> On Fri, Sep 5, 2014 at 1:02 AM, shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >>>>> yep. that's exactly the behavior i saw earlier, and will be figuring >>>>> out first thing tomorrow morning. i bet it's an environment issues on the >>>>> slaves. >>>>> >>>>> >>>>> On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas < >>>>> nicholas.cham...@gmail.com> wrote: >>>>> >>>>>> Looks like during the last build >>>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console> >>>>>> Jenkins was unable to execute a git fetch? >>>>>> >>>>>> >>>>>> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >>>>>>> i'm going to restart jenkins and see if that fixes things. >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp <skn...@berkeley.edu> >>>>>>> wrote: >>>>>>> >>>>>>>> looking >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < >>>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>>> >>>>>>>>> It appears that our main man is having trouble >>>>>>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> >>>>>>>>> hearing new requests >>>>>>>>> <https://github.com/apache/spark/pull/2277#issuecomment-54549106>. >>>>>>>>> >>>>>>>>> Do we need some smelling salts? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp <skn...@berkeley.edu> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> i'd ping the Jenkinsmench... the master was completely offline, >>>>>>>>>> so any new >>>>>>>>>> jobs wouldn't have reached it. any jobs that were queued when >>>>>>>>>> power was >>>>>>>>>> lost probably started up, but jobs that were running would fail. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas < >>>>>>>>>> nicholas.cham...@gmail.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>> > Woohoo! Thanks Shane. >>>>>>>>>> > >>>>>>>>>> > Do you know if queued PR builds will automatically be picked >>>>>>>>>> up? Or do we >>>>>>>>>> > have to ping the Jenkinmensch manually from each PR? >>>>>>>>>> > >>>>>>>>>> > Nick >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp < >>>>>>>>>> skn...@berkeley.edu> wrote: >>>>>>>>>> > >>>>>>>>>> >> AND WE'RE UP! >>>>>>>>>> >> >>>>>>>>>> >> sorry that this took so long... i'll send out a more detailed >>>>>>>>>> explanation >>>>>>>>>> >> of what happened soon. >>>>>>>>>> >> >>>>>>>>>> >> now, off to back up jenkins. >>>>>>>>>> >> >>>>>>>>>> >> shane >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp < >>>>>>>>>> skn...@berkeley.edu> wrote: >>>>>>>>>> >> >>>>>>>>>> >> > it's a faulty power switch on the firewall, which has been >>>>>>>>>> swapped out. >>>>>>>>>> >> > we're about to reboot and be good to go. >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp < >>>>>>>>>> skn...@berkeley.edu> >>>>>>>>>> >> wrote: >>>>>>>>>> >> > >>>>>>>>>> >> >> looks like some hardware failed, and we're swapping in a >>>>>>>>>> replacement. >>>>>>>>>> >> i >>>>>>>>>> >> >> don't have more specific information yet -- including >>>>>>>>>> *what* failed, >>>>>>>>>> >> as our >>>>>>>>>> >> >> sysadmin is super busy ATM. the root cause was an >>>>>>>>>> incorrect circuit >>>>>>>>>> >> being >>>>>>>>>> >> >> switched off during building maintenance. >>>>>>>>>> >> >> >>>>>>>>>> >> >> on a side note, this incident will be accelerating our plan >>>>>>>>>> to move the >>>>>>>>>> >> >> entire jenkins infrastructure in to a managed datacenter >>>>>>>>>> environment. >>>>>>>>>> >> this >>>>>>>>>> >> >> will be our major push over the next couple of weeks. more >>>>>>>>>> details >>>>>>>>>> >> about >>>>>>>>>> >> >> this, also, as soon as i get them. >>>>>>>>>> >> >> >>>>>>>>>> >> >> i'm very sorry about the downtime, we'll get everything up >>>>>>>>>> and running >>>>>>>>>> >> >> ASAP. >>>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp < >>>>>>>>>> skn...@berkeley.edu> >>>>>>>>>> >> wrote: >>>>>>>>>> >> >> >>>>>>>>>> >> >>> looks like a power outage in soda hall. more updates as >>>>>>>>>> they happen. >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp < >>>>>>>>>> skn...@berkeley.edu> >>>>>>>>>> >> >>> wrote: >>>>>>>>>> >> >>> >>>>>>>>>> >> >>>> i am trying to get things up and running, but it looks >>>>>>>>>> like either >>>>>>>>>> >> the >>>>>>>>>> >> >>>> firewall gateway or jenkins server itself is down. i'll >>>>>>>>>> update as >>>>>>>>>> >> soon as >>>>>>>>>> >> >>>> i know more. >>>>>>>>>> >> >>>> >>>>>>>>>> >> >>> >>>>>>>>>> >> >>> >>>>>>>>>> >> >> >>>>>>>>>> >> > >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>>> Google Groups >>>>>>>>>> > "amp-infra" group. >>>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>>> it, send an >>>>>>>>>> > email to amp-infra+unsubscr...@googlegroups.com. >>>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
