Looks like Jenkins is back! lol The poor guy has like a million builds <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> to catch up on.
On Fri, Sep 5, 2014 at 4:15 PM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > How's it going? > > It looks like during the last build > <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/lastBuild/console> > from about 30 min ago Jenkins was still having trouble fetching from > GitHub. It also looks like not all requests for testing are triggering > builds. > > > On Fri, Sep 5, 2014 at 1:23 PM, shane knapp <skn...@berkeley.edu> wrote: > >> it's looking like everything except the pull request builders are >> working. i'm going to be working on getting this resolved today. >> >> >> On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> Hmm, looks like at least some builds >>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull> >>> are working now, though this last one was from ~5 hours ago. >>> >>> >>> On Fri, Sep 5, 2014 at 1:02 AM, shane knapp <skn...@berkeley.edu> wrote: >>> >>>> yep. that's exactly the behavior i saw earlier, and will be figuring >>>> out first thing tomorrow morning. i bet it's an environment issues on the >>>> slaves. >>>> >>>> >>>> On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas < >>>> nicholas.cham...@gmail.com> wrote: >>>> >>>>> Looks like during the last build >>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console> >>>>> Jenkins was unable to execute a git fetch? >>>>> >>>>> >>>>> On Thu, Sep 4, 2014 at 7:58 PM, shane knapp <skn...@berkeley.edu> >>>>> wrote: >>>>> >>>>>> i'm going to restart jenkins and see if that fixes things. >>>>>> >>>>>> >>>>>> On Thu, Sep 4, 2014 at 4:56 PM, shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >>>>>>> looking >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas < >>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>> >>>>>>>> It appears that our main man is having trouble >>>>>>>> <https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/> >>>>>>>> hearing new requests >>>>>>>> <https://github.com/apache/spark/pull/2277#issuecomment-54549106>. >>>>>>>> >>>>>>>> Do we need some smelling salts? >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 4, 2014 at 5:49 PM, shane knapp <skn...@berkeley.edu> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> i'd ping the Jenkinsmench... the master was completely offline, >>>>>>>>> so any new >>>>>>>>> jobs wouldn't have reached it. any jobs that were queued when >>>>>>>>> power was >>>>>>>>> lost probably started up, but jobs that were running would fail. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas < >>>>>>>>> nicholas.cham...@gmail.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>> > Woohoo! Thanks Shane. >>>>>>>>> > >>>>>>>>> > Do you know if queued PR builds will automatically be picked up? >>>>>>>>> Or do we >>>>>>>>> > have to ping the Jenkinmensch manually from each PR? >>>>>>>>> > >>>>>>>>> > Nick >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Thu, Sep 4, 2014 at 5:37 PM, shane knapp <skn...@berkeley.edu> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> >> AND WE'RE UP! >>>>>>>>> >> >>>>>>>>> >> sorry that this took so long... i'll send out a more detailed >>>>>>>>> explanation >>>>>>>>> >> of what happened soon. >>>>>>>>> >> >>>>>>>>> >> now, off to back up jenkins. >>>>>>>>> >> >>>>>>>>> >> shane >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> On Thu, Sep 4, 2014 at 1:27 PM, shane knapp < >>>>>>>>> skn...@berkeley.edu> wrote: >>>>>>>>> >> >>>>>>>>> >> > it's a faulty power switch on the firewall, which has been >>>>>>>>> swapped out. >>>>>>>>> >> > we're about to reboot and be good to go. >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > On Thu, Sep 4, 2014 at 1:19 PM, shane knapp < >>>>>>>>> skn...@berkeley.edu> >>>>>>>>> >> wrote: >>>>>>>>> >> > >>>>>>>>> >> >> looks like some hardware failed, and we're swapping in a >>>>>>>>> replacement. >>>>>>>>> >> i >>>>>>>>> >> >> don't have more specific information yet -- including *what* >>>>>>>>> failed, >>>>>>>>> >> as our >>>>>>>>> >> >> sysadmin is super busy ATM. the root cause was an incorrect >>>>>>>>> circuit >>>>>>>>> >> being >>>>>>>>> >> >> switched off during building maintenance. >>>>>>>>> >> >> >>>>>>>>> >> >> on a side note, this incident will be accelerating our plan >>>>>>>>> to move the >>>>>>>>> >> >> entire jenkins infrastructure in to a managed datacenter >>>>>>>>> environment. >>>>>>>>> >> this >>>>>>>>> >> >> will be our major push over the next couple of weeks. more >>>>>>>>> details >>>>>>>>> >> about >>>>>>>>> >> >> this, also, as soon as i get them. >>>>>>>>> >> >> >>>>>>>>> >> >> i'm very sorry about the downtime, we'll get everything up >>>>>>>>> and running >>>>>>>>> >> >> ASAP. >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> On Thu, Sep 4, 2014 at 12:27 PM, shane knapp < >>>>>>>>> skn...@berkeley.edu> >>>>>>>>> >> wrote: >>>>>>>>> >> >> >>>>>>>>> >> >>> looks like a power outage in soda hall. more updates as >>>>>>>>> they happen. >>>>>>>>> >> >>> >>>>>>>>> >> >>> >>>>>>>>> >> >>> On Thu, Sep 4, 2014 at 12:25 PM, shane knapp < >>>>>>>>> skn...@berkeley.edu> >>>>>>>>> >> >>> wrote: >>>>>>>>> >> >>> >>>>>>>>> >> >>>> i am trying to get things up and running, but it looks >>>>>>>>> like either >>>>>>>>> >> the >>>>>>>>> >> >>>> firewall gateway or jenkins server itself is down. i'll >>>>>>>>> update as >>>>>>>>> >> soon as >>>>>>>>> >> >>>> i know more. >>>>>>>>> >> >>>> >>>>>>>>> >> >>> >>>>>>>>> >> >>> >>>>>>>>> >> >> >>>>>>>>> >> > >>>>>>>>> >> >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > You received this message because you are subscribed to the >>>>>>>>> Google Groups >>>>>>>>> > "amp-infra" group. >>>>>>>>> > To unsubscribe from this group and stop receiving emails from >>>>>>>>> it, send an >>>>>>>>> > email to amp-infra+unsubscr...@googlegroups.com. >>>>>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
