Issues with Kafka PR builds since Jenkins emergency restart
Hi,
As explained in INFRA-14011[1], we're having issues with Kafka PR builds
since the recent Jenkins restart.
Builds hang for a long time with the message:
"GitHub pull request #2301 of commit
0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
This is problematic for Kafka (we generally only merge PRs when builds are
green), but also for other projects because these builds are taking a lot
of executor slots for long periods of time and not doing anything.
In one of the builds that I interrupted, there were some errors related to
API rate limits:
Unable to query GitHub for status of
PullRequestjava.io.InterruptedIOException
at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandler.java:41)
at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
at org.kohsuke.github.Requester._to(Requester.java:283)
at org.kohsuke.github.Requester.to(Requester.java:231)
at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuilds.java:101)
at
org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:18)
at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
exceeded for asfbot.","documentation_url":"
https://developer.github.com/v3/#rate-limiting"}
at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
... 11 more
Caused by: org.kohsuke.github.HttpException: Server returned HTTP response
code: 403, message: 'Forbidden' for URL:
https://api.github.com/repos/apache/kafka/pulls/2910
at org.kohsuke.github.Requester.parse(Requester.java:612)
at org.kohsuke.github.Requester._to(Requester.java:262)
... 10 more
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: https://api.github.com/repos/apache/kafka/pulls/2910
at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.kohsuke.github.Requester.parse(Requester.java:596)
... 11 more
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: https://api.github.com/repos/apache/kafka/pulls/2910
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at org.kohsuke.github.Requester.parse(Requester.java:586)
... 11 more
Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING with
url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
message: 'Build started sha1 is original commit.'[3]
Could this be a hint of the underlying issue?
These problems are affecting the following jobs:
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
Thanks for your help.
Ismael
[1] https://issues.apache.org/jira/browse/INFRA-14011
[2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
[3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console
Re: Issues with Kafka PR builds since Jenkins emergency restart
Hi,
Yes, we have contacted Github Support about the rate limit, and are yet to hear
back.
Will let you know when we hear something.
Gav… (ASF Infra)
> On 27 Apr 2017, at 8:16 pm, Ismael Juma wrote:
>
> Hi,
>
> As explained in INFRA-14011[1], we're having issues with Kafka PR builds
> since the recent Jenkins restart.
>
> Builds hang for a long time with the message:
>
> "GitHub pull request #2301 of commit
> 0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
>
> This is problematic for Kafka (we generally only merge PRs when builds are
> green), but also for other projects because these builds are taking a lot
> of executor slots for long periods of time and not doing anything.
>
> In one of the builds that I interrupted, there were some errors related to
> API rate limits:
>
> Unable to query GitHub for status of
> PullRequestjava.io.InterruptedIOException
> at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandler.java:41)
> at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
> at org.kohsuke.github.Requester._to(Requester.java:283)
> at org.kohsuke.github.Requester.to(Requester.java:231)
> at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
> at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
> at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(GhprbBuilds.java:101)
> at
> org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(GhprbBuildListener.java:18)
> at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
> at hudson.model.Run.execute(Run.java:1724)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:404)
> Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
> exceeded for asfbot.","documentation_url":"
> https://developer.github.com/v3/#rate-limiting"}
> at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
> ... 11 more
> Caused by: org.kohsuke.github.HttpException: Server returned HTTP response
> code: 403, message: 'Forbidden' for URL:
> https://api.github.com/repos/apache/kafka/pulls/2910
> at org.kohsuke.github.Requester.parse(Requester.java:612)
> at org.kohsuke.github.Requester._to(Requester.java:262)
> ... 10 more
> Caused by: java.io.IOException: Server returned HTTP response code: 403 for
> URL: https://api.github.com/repos/apache/kafka/pulls/2910
> at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1926)
> at
> sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1921)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1920)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1490)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
> at org.kohsuke.github.Requester.parse(Requester.java:596)
> ... 11 more
> Caused by: java.io.IOException: Server returned HTTP response code: 403 for
> URL: https://api.github.com/repos/apache/kafka/pulls/2910
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1876)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
> at org.kohsuke.github.Requester.parse(Requester.java:586)
> ... 11 more
> Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING with
> url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
> message: 'Build started sha1 is original commit.'[3]
>
> Could this be a hint of the underlying issue?
>
> These problems are affecting the following jobs:
>
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
>
> Thanks for your help.
>
> Ismael
>
> [1] https://issues.apache.org/jira/browse/INFRA-14011
> [2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
> [3] https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/console
Re: Issues with Kafka PR builds since Jenkins emergency restart
Thanks!
Ismael
On Thu, Apr 27, 2017 at 11:53 AM, Gavin McDonald
wrote:
> Hi,
>
> Yes, we have contacted Github Support about the rate limit, and are yet to
> hear back.
>
> Will let you know when we hear something.
>
> Gav… (ASF Infra)
>
> > On 27 Apr 2017, at 8:16 pm, Ismael Juma wrote:
> >
> > Hi,
> >
> > As explained in INFRA-14011[1], we're having issues with Kafka PR builds
> > since the recent Jenkins restart.
> >
> > Builds hang for a long time with the message:
> >
> > "GitHub pull request #2301 of commit
> > 0cc900288e61b05b4095b63d28969e16167e5d88, no merge conflicts."[2]
> >
> > This is problematic for Kafka (we generally only merge PRs when builds
> are
> > green), but also for other projects because these builds are taking a lot
> > of executor slots for long periods of time and not doing anything.
> >
> > In one of the builds that I interrupted, there were some errors related
> to
> > API rate limits:
> >
> > Unable to query GitHub for status of
> > PullRequestjava.io.InterruptedIOException
> > at org.kohsuke.github.RateLimitHandler$1.onError(
> RateLimitHandler.java:41)
> > at org.kohsuke.github.Requester.handleApiError(Requester.java:666)
> > at org.kohsuke.github.Requester._to(Requester.java:283)
> > at org.kohsuke.github.Requester.to(Requester.java:231)
> > at org.kohsuke.github.GHPullRequest.populate(GHPullRequest.java:207)
> > at org.kohsuke.github.GHPullRequest.getMergeable(GHPullRequest.java:170)
> > at org.jenkinsci.plugins.ghprb.GhprbBuilds.onStarted(
> GhprbBuilds.java:101)
> > at
> > org.jenkinsci.plugins.ghprb.GhprbBuildListener.onStarted(
> GhprbBuildListener.java:18)
> > at hudson.model.listeners.RunListener.fireStarted(RunListener.java:240)
> > at hudson.model.Run.execute(Run.java:1724)
> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> > at hudson.model.ResourceController.execute(ResourceController.java:98)
> > at hudson.model.Executor.run(Executor.java:404)
> > Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
> > exceeded for asfbot.","documentation_url":"
> > https://developer.github.com/v3/#rate-limiting"}
> > at org.kohsuke.github.Requester.handleApiError(Requester.java:654)
> > ... 11 more
> > Caused by: org.kohsuke.github.HttpException: Server returned HTTP
> response
> > code: 403, message: 'Forbidden' for URL:
> > https://api.github.com/repos/apache/kafka/pulls/2910
> > at org.kohsuke.github.Requester.parse(Requester.java:612)
> > at org.kohsuke.github.Requester._to(Requester.java:262)
> > ... 10 more
> > Caused by: java.io.IOException: Server returned HTTP response code: 403
> for
> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
> > at sun.reflect.GeneratedConstructorAccessor241.newInstance(Unknown
> Source)
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > at
> > sun.net.www.protocol.http.HttpURLConnection$10.run(
> HttpURLConnection.java:1926)
> > at
> > sun.net.www.protocol.http.HttpURLConnection$10.run(
> HttpURLConnection.java:1921)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getChainedException(
> HttpURLConnection.java:1920)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1490)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:1474)
> > at
> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(
> HttpsURLConnectionImpl.java:254)
> > at org.kohsuke.github.Requester.parse(Requester.java:596)
> > ... 11 more
> > Caused by: java.io.IOException: Server returned HTTP response code: 403
> for
> > URL: https://api.github.com/repos/apache/kafka/pulls/2910
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1876)
> > at
> > sun.net.www.protocol.http.HttpURLConnection.getInputStream(
> HttpURLConnection.java:1474)
> > at java.net.HttpURLConnection.getResponseCode(
> HttpURLConnection.java:480)
> > at
> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(
> HttpsURLConnectionImpl.java:338)
> > at org.kohsuke.github.Requester.parse(Requester.java:586)
> > ... 11 more
> > Setting status of b5df4d319bc65e847a8c18ad20660f3e506e9a16 to PENDING
> with
> > url https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3201/ and
> > message: 'Build started sha1 is original commit.'[3]
> >
> > Could this be a hint of the underlying issue?
> >
> > These problems are affecting the following jobs:
> >
> > https://builds.apache.org/job/kafka-pr-jdk7-scala2.10
> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.11
> > https://builds.apache.org/job/kafka-pr-jdk8-scala2.12
> >
> > Thanks for your help.
> >
> > Ismael
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-14011
> > [2] https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3201/console
> >
