I have a problem with a with Foreman and Puppet server (6.2.1) on a KVM VM with Ubuntu 18.04. After a crash of the VM Foreman starts without a problem (no errors in it's logs) but puppetserver refuses with this JRuby java error below. Quite frankly I have no clue as to where to start looking.
Anyone has any experience with this kind of error? 2019-10-09T09:18:02.909+02:00 INFO [async-dispatch-2] [p.s.v.versioned-code-service] No code-id-command set for versioned-code-service. Code-id will be nil. 2019-10-09T09:18:02.910+02:00 INFO [async-dispatch-2] [p.s.v.versioned-code-service] No code-content-command set for versioned-code-service. Attempting to fetch code content will fail. 2019-10-09T09:18:12.148+02:00 ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception! java.lang.IllegalStateException: There was a problem adding a JRubyInstance to the pool. at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32598$prime_pool_BANG___32603$fn__32607.invoke(jruby_agents.clj:75) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32598$prime_pool_BANG___32603.invoke(jruby_agents.clj:48) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32848$send_prime_pool_BANG___32853$fn__32854$fn__32855.invoke(jruby_agents.clj:233) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:389) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:374) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364) at puppetlabs.trapperkeeper.internal$fn__14006$shutdown_service__14011$fn$reify__14013$service_fnk__4991__auto___positional$reify__14018.shutdown_on_error(internal.clj:429) at puppetlabs.trapperkeeper.internal$fn__13953$G__13938__13961.invoke(internal.clj:397) at puppetlabs.trapperkeeper.internal$fn__13953$G__13937__13970.invoke(internal.clj:397) at clojure.core$partial$fn__5824.invoke(core.clj:2625) at clojure.core$partial$fn__5824.invoke(core.clj:2624) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32573$send_agent__32578$fn__32579$agent_fn__32580.invoke(jruby_agents.clj:42) at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2033) at clojure.lang.AFn.applyToHelper(AFn.java:154) at clojure.lang.RestFn.applyTo(RestFn.java:132) at clojure.lang.Agent$Action.doRun(Agent.java:114) at clojure.lang.Agent$Action.run(Agent.java:163) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- concurrent at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:131) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1295) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) at puppetlabs.services.jruby.jruby_puppet_core$fn__34109$get_initialize_pool_instance_fn__34114$fn__34115$fn__34116.invoke(jruby_puppet_core.clj:132) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32180$create_pool_instance_BANG___32189$fn__32192.invoke(jruby_internal.clj:211) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32180$create_pool_instance_BANG___32189.invoke(jruby_internal.clj:177) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32598$prime_pool_BANG___32603$fn__32607.invoke(jruby_agents.clj:65) ... 20 common frames omitted Caused by: org.jruby.exceptions.LoadError: (LoadError) no such file to load -- concurrent at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/thread_local.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<module:(root)>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:1) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.Puppet(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:41) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:38) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1) at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.(root)(uri:classloader:/puppetserver-lib/puppet/server.rb:1) at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1) 2019-10-09T09:18:12.150+02:00 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!! java.lang.IllegalStateException: Unable to borrow JRubyInstance from pool at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32380$borrow_from_pool_BANG__STAR___32385$fn__32386.invoke(jruby_internal.clj:268) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32380$borrow_from_pool_BANG__STAR___32385.invoke(jruby_internal.clj:255) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32427$borrow_from_pool_with_timeout__32432$fn__32433.invoke(jruby_internal.clj:303) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32427$borrow_from_pool_with_timeout__32432.invoke(jruby_internal.clj:292) at puppetlabs.services.jruby_pool_manager.jruby_core$fn__33782$borrow_from_pool_with_timeout__33787$fn__33788.invoke(jruby_core.clj:210) at puppetlabs.services.jruby_pool_manager.jruby_core$fn__33782$borrow_from_pool_with_timeout__33787.invoke(jruby_core.clj:197) at puppetlabs.services.config.puppet_server_config_core$fn__40533$get_puppet_config__40538$fn__40539$fn__40540.invoke(puppet_server_config_core.clj:106) at puppetlabs.services.config.puppet_server_config_core$fn__40533$get_puppet_config__40538$fn__40539.invoke(puppet_server_config_core.clj:106) at puppetlabs.services.config.puppet_server_config_core$fn__40533$get_puppet_config__40538.invoke(puppet_server_config_core.clj:101) at puppetlabs.services.config.puppet_server_config_service$reify__40568$service_fnk__4991__auto___positional$reify__40579.init(puppet_server_config_service.clj:25) at puppetlabs.trapperkeeper.services$fn__4815$G__4807__4818.invoke(services.clj:9) at puppetlabs.trapperkeeper.services$fn__4815$G__4806__4822.invoke(services.clj:9) at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560$fn__13561.invoke(internal.clj:198) at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560.invoke(internal.clj:181) at puppetlabs.trapperkeeper.internal$fn__13582$run_lifecycle_fns__13587$fn__13588.invoke(internal.clj:231) at puppetlabs.trapperkeeper.internal$fn__13582$run_lifecycle_fns__13587.invoke(internal.clj:208) at puppetlabs.trapperkeeper.internal$fn__14155$build_app_STAR___14164$fn$reify__14176.init(internal.clj:583) at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210$fn__14211$fn__14213.invoke(internal.clj:611) at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210$fn__14211.invoke(internal.clj:610) at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210.invoke(internal.clj:604) at clojure.core$partial$fn__5826.invoke(core.clj:2630) at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817.invoke(internal.clj:251) at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814.invoke(internal.clj:251) at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973) at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972) at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977) at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975) at clojure.core.async$ioc_alts_BANG_$fn__11006.invoke(async.clj:384) at clojure.core.async$do_alts$fn__10946$fn__10949.invoke(async.clj:253) at clojure.core.async.impl.channels.ManyToManyChannel$fn__6246$fn__6247.invoke(channels.clj:95) at clojure.lang.AFn.run(AFn.java:22) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- concurrent at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:131) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1295) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) at puppetlabs.services.jruby.jruby_puppet_core$fn__34109$get_initialize_pool_instance_fn__34114$fn__34115$fn__34116.invoke(jruby_puppet_core.clj:132) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32180$create_pool_instance_BANG___32189$fn__32192.invoke(jruby_internal.clj:211) at puppetlabs.services.jruby_pool_manager.impl.jruby_internal$fn__32180$create_pool_instance_BANG___32189.invoke(jruby_internal.clj:177) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32598$prime_pool_BANG___32603$fn__32607.invoke(jruby_agents.clj:65) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32598$prime_pool_BANG___32603.invoke(jruby_agents.clj:48) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32848$send_prime_pool_BANG___32853$fn__32854$fn__32855.invoke(jruby_agents.clj:233) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:389) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invokeStatic(internal.clj:374) at puppetlabs.trapperkeeper.internal$shutdown_on_error_STAR_.invoke(internal.clj:364) at puppetlabs.trapperkeeper.internal$fn__14006$shutdown_service__14011$fn$reify__14013$service_fnk__4991__auto___positional$reify__14018.shutdown_on_error(internal.clj:429) at puppetlabs.trapperkeeper.internal$fn__13953$G__13938__13961.invoke(internal.clj:397) at puppetlabs.trapperkeeper.internal$fn__13953$G__13937__13970.invoke(internal.clj:397) at clojure.core$partial$fn__5824.invoke(core.clj:2625) at clojure.core$partial$fn__5824.invoke(core.clj:2624) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32573$send_agent__32578$fn__32579$agent_fn__32580.invoke(jruby_agents.clj:42) at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2033) at clojure.lang.AFn.applyToHelper(AFn.java:154) at clojure.lang.RestFn.applyTo(RestFn.java:132) at clojure.lang.Agent$Action.doRun(Agent.java:114) at clojure.lang.Agent$Action.run(Agent.java:163) ... 3 common frames omitted Caused by: org.jruby.exceptions.LoadError: (LoadError) no such file to load -- concurrent at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/thread_local.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.rubygems.core_ext.kernel_require.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at RUBY.<module:(root)>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:1) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:1) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.Puppet(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:41) at RUBY.<main>(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:38) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1) at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:59) at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:970) at RUBY.(root)(uri:classloader:/puppetserver-lib/puppet/server.rb:1) at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1) 2019-10-09T09:18:12.173+02:00 INFO [main] [p.t.internal] Beginning shutdown sequence 2019-10-09T09:18:12.212+02:00 INFO [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopping metrics sampler job 2019-10-09T09:18:12.213+02:00 INFO [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopped metrics sampler job 2019-10-09T09:18:12.221+02:00 INFO [async-dispatch-3] [p.s.j.i.jruby-agents] Draining JRuby pool. 2019-10-09T09:18:12.226+02:00 ERROR [async-dispatch-3] [p.t.internal] Encountered error during shutdown sequence java.lang.InterruptedException: Lock can't be granted because a pill has been inserted at com.puppetlabs.jruby_utils.pool.JRubyPool.lockWithTimeout(JRubyPool.java:368) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32679$borrow_all_jrubies__32684$fn__32685$fn__32686.invoke(jruby_agents.clj:126) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32679$borrow_all_jrubies__32684$fn__32685.invoke(jruby_agents.clj:125) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32679$borrow_all_jrubies__32684.invoke(jruby_agents.clj:117) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32759$drain_and_refill_pool_BANG___32768$fn__32771.invoke(jruby_agents.clj:191) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32759$drain_and_refill_pool_BANG___32768.invoke(jruby_agents.clj:173) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32800$flush_pool_for_shutdown_BANG___32805$fn__32806.invoke(jruby_agents.clj:212) at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$fn__32800$flush_pool_for_shutdown_BANG___32805.invoke(jruby_agents.clj:200) at puppetlabs.services.jruby_pool_manager.jruby_core$fn__33853$flush_pool_for_shutdown_BANG___33858$fn__33859.invoke(jruby_core.clj:236) at puppetlabs.services.jruby_pool_manager.jruby_core$fn__33853$flush_pool_for_shutdown_BANG___33858.invoke(jruby_core.clj:231) at puppetlabs.services.jruby.jruby_puppet_service$reify__34636$service_fnk__4991__auto___positional$reify__34650.stop(jruby_puppet_service.clj:52) at puppetlabs.trapperkeeper.services$fn__4841$G__4811__4844.invoke(services.clj:9) at puppetlabs.trapperkeeper.services$fn__4841$G__4810__4848.invoke(services.clj:9) at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560$fn__13561.invoke(internal.clj:198) at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560.invoke(internal.clj:181) at puppetlabs.trapperkeeper.internal$fn__14063$shutdown_BANG___14068$fn__14069$shutdown_fn__14071$fn__14086.invoke(internal.clj:440) at puppetlabs.trapperkeeper.internal$fn__14063$shutdown_BANG___14068$fn__14069$shutdown_fn__14071.invoke(internal.clj:439) at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817$fn__13831.invoke(internal.clj:276) at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817.invoke(internal.clj:260) at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814.invoke(internal.clj:251) at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973) at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972) at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977) at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975) at clojure.core.async$ioc_alts_BANG_$fn__11006.invoke(async.clj:384) at clojure.core.async$do_alts$fn__10946$fn__10949.invoke(async.clj:253) at clojure.core.async.impl.channels.ManyToManyChannel$fn__6262.invoke(channels.clj:135) at clojure.lang.AFn.run(AFn.java:22) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-10-09T09:18:12.240+02:00 INFO [async-dispatch-3] [p.t.s.w.jetty9-service] Shutting down web server(s). 2019-10-09T09:18:12.250+02:00 INFO [async-dispatch-3] [p.t.s.s.scheduler-service] Shutting down Scheduler Service -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f20b101a-f8f0-44c0-9671-e4f7cde14b1d%40googlegroups.com.