Looks like Infra has figured it out. From Chris Lambertus on builds.apache.org this morning:
> Looks like this was primarily due to the switch to systemd startup in ubuntu > 16.04 and the need to update DefaultLimitNOFILE in /etc/systemd/system.conf > (grr) to affect processes started via systemctl. Thanks Joan Touzet for > pointing this out. -- Steve www.lucidworks.com > On Jul 18, 2017, at 2:09 PM, Steve Rowe <[email protected]> wrote: > > An update from Chris Lambertus on [email protected] mailing list, about 2 > hours ago: > >> This appears to be a bug in one of the new versions of either jenkins or a >> plugin. We are still looking into the issue. The limit was raised yesterday >> by a factor of 4. > > > -- > Steve > www.lucidworks.com > >> On Jul 18, 2017, at 11:49 AM, Steve Rowe <[email protected]> wrote: >> >> On the [email protected] mailing list last night, on thread “jenkins >> emergency restart notice”, Chris Lambertus wrote: >> >>> Jenkins will be restarted at midnight tonight UTC to address >>> problems related to ‘too many open files’ and to attempt to >>> correct a problem with some windows agents. I will be stopping >>> new builds so that the queue can empty over the next 2 hours. >> >> This morning, two people responded on that thread that builds for the Apache >> Kafka and Apache Brooklyn projects are still encountering this problem. >> >> I’ll keep an eye on this thread. Hopefully they’ll figure it out shortly. >> It’s possible that our issue is different from that of other projects; if >> the situation isn’t resolved in the next 24 hours I’ll ask on hipchat. >> >> -- >> Steve >> www.lucidworks.com >> >>> On Jul 18, 2017, at 11:28 AM, Cassandra Targett <[email protected]> >>> wrote: >>> >>> This is the 2nd one of these I've seen in 2 days (different jobs) but >>> it seems to resolve itself on the next run. >>> >>> Steve Rowe, do you have any insight into what we can do to avoid it? >>> I'll be happy to ask INFRA if not. >>> >>> On Tue, Jul 18, 2017 at 7:32 AM, Apache Jenkins Server >>> <[email protected]> wrote: >>>> Build: https://builds.apache.org/job/Solr-reference-guide-7.x/156/ >>>> >>>> Log: >>>> Started by timer >>>> [EnvInject] - Loading node environment variables. >>>> Building remotely on H19 (git-websites) in workspace >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x >>>>> git rev-parse --is-inside-work-tree # timeout=10 >>>> Fetching changes from the remote Git repository >>>>> git config remote.origin.url git://git.apache.org/lucene-solr.git # >>>>> timeout=10 >>>> Cleaning workspace >>>>> git rev-parse --verify HEAD # timeout=10 >>>> Resetting working tree >>>>> git reset --hard # timeout=10 >>>>> git clean -fdx # timeout=10 >>>> Fetching upstream changes from git://git.apache.org/lucene-solr.git >>>>> git --version # timeout=10 >>>>> git fetch --tags --progress git://git.apache.org/lucene-solr.git >>>>> +refs/heads/*:refs/remotes/origin/* >>>>> git rev-parse refs/remotes/origin/branch_7x^{commit} # timeout=10 >>>>> git rev-parse refs/remotes/origin/origin/branch_7x^{commit} # timeout=10 >>>> Checking out Revision e66ef6b992c248b66200ac97814951e5d312ab6c >>>> (refs/remotes/origin/branch_7x) >>>> Commit message: "SOLR-11105: Randomize PointFields in "cloud-minimal-jmx" >>>> and "cloud-minimal-inplace-updates" test configsets" >>>>> git config core.sparsecheckout # timeout=10 >>>>> git checkout -f e66ef6b992c248b66200ac97814951e5d312ab6c >>>>> git rev-list e66ef6b992c248b66200ac97814951e5d312ab6c # timeout=10 >>>> No emails were triggered. >>>> [Solr-reference-guide-7.x] $ /bin/bash -xe >>>> /tmp/jenkins8362459747270854181.sh >>>> + set -x >>>> + set -e >>>> + RVM_PATH=/home/jenkins/.rvm >>>> + RUBY_VERSION=ruby-2.3.3 >>>> + GEMSET=solr-refguide-gemset >>>> + curl -sSL https://get.rvm.io >>>> + bash -s -- --ignore-dotfiles stable >>>> Turning on ignore dotfiles mode. >>>> Downloading https://github.com/rvm/rvm/archive/1.29.2.tar.gz >>>> Downloading >>>> https://github.com/rvm/rvm/releases/download/1.29.2/1.29.2.tar.gz.asc >>>> Found PGP signature at: >>>> 'https://github.com/rvm/rvm/releases/download/1.29.2/1.29.2.tar.gz.asc', >>>> but no GPG software exists to validate it, skipping. >>>> >>>> Upgrading the RVM installation in /home/jenkins/.rvm/ >>>> Upgrade of RVM in /home/jenkins/.rvm/ is complete. >>>> >>>> # jenkins, >>>> # >>>> # Thank you for using RVM! >>>> # We sincerely hope that RVM helps to make your life easier and more >>>> enjoyable!!! >>>> # >>>> # ~Wayne, Michal & team. >>>> >>>> In case of problems: https://rvm.io/help and https://twitter.com/rvm_io >>>> >>>> Upgrade Notes: >>>> >>>> * No new notes to display. >>>> >>>> + set +x >>>> Running 'source /home/jenkins/.rvm/scripts/rvm' >>>> Running 'rvm autolibs disable' >>>> Running 'rvm install ruby-2.3.3' >>>> Already installed ruby-2.3.3. >>>> To reinstall use: >>>> >>>> rvm reinstall ruby-2.3.3 >>>> >>>> Running 'rvm gemset create solr-refguide-gemset' >>>> ruby-2.3.3 - #gemset created >>>> /home/jenkins/.rvm/gems/ruby-2.3.3@solr-refguide-gemset >>>> ruby-2.3.3 - #generating solr-refguide-gemset wrappers........ >>>> Running 'rvm ruby-2.3.3@solr-refguide-gemset' >>>> Using /home/jenkins/.rvm/gems/ruby-2.3.3 with gemset solr-refguide-gemset >>>> Running 'gem install --force --version 3.5.0 jekyll' >>>> Successfully installed jekyll-3.5.0 >>>> Parsing documentation for jekyll-3.5.0 >>>> Done installing documentation for jekyll after 2 seconds >>>> 1 gem installed >>>> Running 'gem install --force --version 2.1.0 jekyll-asciidoc' >>>> Successfully installed jekyll-asciidoc-2.1.0 >>>> Parsing documentation for jekyll-asciidoc-2.1.0 >>>> Done installing documentation for jekyll-asciidoc after 0 seconds >>>> 1 gem installed >>>> Running 'gem install --force --version 1.1.2 pygments.rb' >>>> Successfully installed pygments.rb-1.1.2 >>>> Parsing documentation for pygments.rb-1.1.2 >>>> Done installing documentation for pygments.rb after 0 seconds >>>> 1 gem installed >>>> + ant clean build-site build-pdf >>>> Buildfile: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/solr-ref-guide/build.xml >>>> >>>> clean: >>>> >>>> build-init: >>>> [mkdir] Created dir: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [echo] Copying all non template files from src ... >>>> [copy] Copying 343 files to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [echo] Copy (w/prop replacement) any template files from src... >>>> [copy] Copying 1 file to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> >>>> ivy-availability-check: >>>> >>>> ivy-fail: >>>> >>>> ivy-configure: >>>> [ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: >>>> http://ant.apache.org/ivy/ :: >>>> [ivy:configure] :: loading settings :: file = >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/lucene/top-level-ivy-settings.xml >>>> >>>> resolve: >>>> [ivy:retrieve] >>>> [ivy:retrieve] :: problems summary :: >>>> [ivy:retrieve] :::: ERRORS >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] unknown resolver null >>>> [ivy:retrieve] >>>> [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS >>>> >>>> build-tools-jar: >>>> [mkdir] Created dir: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/classes >>>> [javac] Compiling 3 source files to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/classes >>>> [jar] Building jar: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/solr-ref-guide-tools.jar >>>> >>>> build-nav-data-files: >>>> [java] Building up tree of all known pages >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/pdf-main-body.adoc >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/scrollnav.json >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/sidebar.json >>>> >>>> -build-site: >>>> [echo] Running Jekyll... >>>> [exec] Configuration file: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_config.yml >>>> [exec] Source: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [exec] Destination: ../html-site >>>> [exec] Incremental build: disabled. Enable with --incremental >>>> [exec] Generating... >>>> [exec] [31m Deprecation: The 'gems' configuration option has been >>>> renamed to 'plugins'. Please update your config file accordingly. [0m >>>> [exec] done in 15.003 seconds. >>>> [exec] Auto-regeneration: disabled. Use --watch to enable. >>>> >>>> check-links-and-anchors: >>>> >>>> build-site: >>>> [echo] Ready to browse site: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/html-site/index.html >>>> >>>> build-init: >>>> [delete] Deleting directory >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [mkdir] Created dir: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [echo] Copying all non template files from src ... >>>> [copy] Copying 343 files to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> [echo] Copy (w/prop replacement) any template files from src... >>>> [copy] Copying 1 file to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content >>>> >>>> ivy-availability-check: >>>> >>>> ivy-configure: >>>> [ivy:configure] :: loading settings :: file = >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/lucene/top-level-ivy-settings.xml >>>> >>>> resolve: >>>> >>>> build-tools-jar: >>>> >>>> build-nav-data-files: >>>> [java] Building up tree of all known pages >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/pdf-main-body.adoc >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/scrollnav.json >>>> [java] Creating >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/_data/sidebar.json >>>> >>>> -build-raw-pdf: >>>> [mkdir] Created dir: >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/pdf-tmp >>>> [asciidoctor:convert] Using inherited external environment to resolve gems >>>> (/home/jenkins/.rvm/gems/ruby-2.3.3@solr-refguide-gemset), i.e. build is >>>> platform dependent! >>>> [asciidoctor:convert] Render SolrRefGuide-all.adoc from >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/content/pdf >>>> to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/pdf-tmp >>>> with backend=pdf >>>> [asciidoctor:convert] Ignoring executable-hooks-1.3.2 because its >>>> extensions are not built. Try: gem pristine executable-hooks --version >>>> 1.3.2 >>>> [asciidoctor:convert] Ignoring ffi-1.9.18 because its extensions are not >>>> built. Try: gem pristine ffi --version 1.9.18 >>>> [asciidoctor:convert] Ignoring gem-wrappers-1.2.7 because its extensions >>>> are not built. Try: gem pristine gem-wrappers --version 1.2.7 >>>> [asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: >>>> invalid part, must have at least one section (e.g., chapter, appendix, >>>> etc.) >>>> [asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: >>>> invalid part, must have at least one section (e.g., chapter, appendix, >>>> etc.) >>>> [move] Moving 1 file to >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/pdf-tmp >>>> >>>> -reduce-pdf-size: >>>> >>>> build-pdf: >>>> [echo] Finished Building >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/apache-solr-ref-guide-7.1-DRAFT.pdf >>>> >>>> BUILD SUCCESSFUL >>>> Total time: 3 minutes 43 seconds >>>> Archiving artifacts >>>> ERROR: Failed to archive artifacts: solr/build/solr-ref-guide/*.pdf >>>> java.io.IOException: Too many open files >>>> at java.io.UnixFileSystem.createFileExclusively(Native Method) >>>> at java.io.File.createTempFile(File.java:2024) >>>> at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:68) >>>> Caused: java.io.IOException: Failed to create a temporary file in >>>> /x1/jenkins/jenkins-home/jobs/Solr-reference-guide-7.x/builds/156 >>>> at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:70) >>>> at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:57) >>>> at hudson.XmlFile.write(XmlFile.java:168) >>>> at hudson.model.Run.save(Run.java:1923) >>>> at hudson.model.Run.pickArtifactManager(Run.java:1065) >>>> at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:244) >>>> at >>>> hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) >>>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676) >>>> at hudson.model.Build$BuildExecution.post2(Build.java:186) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621) >>>> at hudson.model.Run.execute(Run.java:1760) >>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>> at hudson.model.Executor.run(Executor.java:405) >>>> Build step 'Archive the artifacts' changed build result to FAILURE >>>> Publishing Javadoc >>>> FATAL: Unable to copy Javadoc from >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/html-site >>>> to >>>> /x1/jenkins/jenkins-home/jobs/Solr-reference-guide-7.x/builds/156/javadoc >>>> java.io.IOException: java.nio.file.FileSystemException: >>>> /x1/jenkins/jenkins-home/jobs/Solr-reference-guide-7.x/builds/156/javadoc/README.md: >>>> Too many open files >>>> at >>>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) >>>> at >>>> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) >>>> at >>>> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) >>>> at >>>> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) >>>> at >>>> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) >>>> at java.nio.file.Files.newOutputStream(Files.java:216) >>>> at hudson.util.IOUtils.copy(IOUtils.java:39) >>>> at hudson.FilePath.readFromTar(FilePath.java:2287) >>>> Caused: java.io.IOException: Failed to extract >>>> /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-7.x/solr/build/solr-ref-guide/html-site/**/* >>>> at hudson.FilePath.readFromTar(FilePath.java:2297) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2207) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2115) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2098) >>>> at hudson.tasks.JavadocArchiver.perform(JavadocArchiver.java:108) >>>> at >>>> hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) >>>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676) >>>> at hudson.model.Build$BuildExecution.post2(Build.java:186) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621) >>>> at hudson.model.Run.execute(Run.java:1760) >>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>> at hudson.model.Executor.run(Executor.java:405) >>>> >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2214) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2115) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2098) >>>> at hudson.tasks.JavadocArchiver.perform(JavadocArchiver.java:108) >>>> at >>>> hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) >>>> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676) >>>> at hudson.model.Build$BuildExecution.post2(Build.java:186) >>>> at >>>> hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621) >>>> at hudson.model.Run.execute(Run.java:1760) >>>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>> at hudson.model.Executor.run(Executor.java:405) >>>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: >>>> This archives contains unclosed entries. >>>> at hudson.remoting.Channel$2.adapt(Channel.java:862) >>>> at hudson.remoting.Channel$2.adapt(Channel.java:857) >>>> at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) >>>> at hudson.FilePath.copyRecursiveTo(FilePath.java:2210) >>>> ... 13 more >>>> Caused by: java.io.IOException: This archives contains unclosed entries. >>>> at >>>> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225) >>>> at >>>> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241) >>>> at hudson.util.io.TarArchiver.close(TarArchiver.java:124) >>>> at hudson.FilePath.writeToTar(FilePath.java:2261) >>>> at hudson.FilePath.access$2100(FilePath.java:196) >>>> at hudson.FilePath$45.invoke(FilePath.java:2202) >>>> at hudson.FilePath$45.invoke(FilePath.java:2198) >>>> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2739) >>>> at hudson.remoting.UserRequest.perform(UserRequest.java:153) >>>> at hudson.remoting.UserRequest.perform(UserRequest.java:50) >>>> at hudson.remoting.Request$2.run(Request.java:336) >>>> at >>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:748) >>>> at ......remote call to H19(Native Method) >>>> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545) >>>> at hudson.remoting.UserResponse.retrieve(UserRequest.java:253) >>>> at hudson.remoting.Channel$2.adapt(Channel.java:860) >>>> ... 16 more >>>> Email was triggered for: Failure - Any >>>> Sending email for trigger: Failure - Any >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
