Build:
https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/51/
2 tests failed.
FAILED: org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test
Error Message:
document count mismatch. control=1638 sum(shards)=1652 cloudClient=1652
Stack Trace:
java.lang.AssertionError: document count mismatch. control=1638
sum(shards)=1652 cloudClient=1652
at
__randomizedtesting.SeedInfo.seed([29E266F0FBEBE94:8ACA19B5A142D36C]:0)
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1499)
at
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:266)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.StressHdfsTest.test
Error Message:
Error from server at http://127.0.0.1:34861/_/l/delete_data_dir: Async
exception during distributed update: Error from server at
http://127.0.0.1:39225/_/l/delete_data_dir_shard2_replica_n2/: null request:
http://127.0.0.1:39225/_/l/delete_data_dir_shard2_replica_n2/ Remote error
message: Server error writing document id 1404 to the index
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:34861/_/l/delete_data_dir: Async exception
during distributed update: Error from server at
http://127.0.0.1:39225/_/l/delete_data_dir_shard2_replica_n2/: null
request: http://127.0.0.1:39225/_/l/delete_data_dir_shard2_replica_n2/
Remote error message: Server error writing document id 1404 to the index
at
__randomizedtesting.SeedInfo.seed([29E266F0FBEBE94:8ACA19B5A142D36C]:0)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:650)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:256)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:177)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:156)
at
org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:193)
at
org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13745 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/init-core-data-001
[junit4] 2> 1219087 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=100 numCloses=100
[junit4] 2> 1219087 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1219095 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via:
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 1219095 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1219096 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /mz/x
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 1219226 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 1219295 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 1219298 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1219351 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1219351 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1219351 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1219371 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@547acdc5{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
[junit4] 2> 1219590 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.w.WebAppContext@7e8477d5{hdfs,/,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-38583-hdfs-_-any-5748904265369993100.dir/webapp/,AVAILABLE}{/hdfs}
[junit4] 2> 1219591 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@51067aeb{HTTP/1.1,[http/1.1]}{localhost:38583}
[junit4] 2> 1219591 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server Started @1219644ms
[junit4] 2> 1219733 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.s.c.MetricsLoggerTask Metrics logging will not be async since the
logger is not log4j
[junit4] 2> 1219798 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 1219802 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1219804 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1219804 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1219804 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1219805 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@2b8e698{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
[junit4] 2> 1220027 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.w.WebAppContext@3ec8acba{datanode,/,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-39173-datanode-_-any-4092390915426280258.dir/webapp/,AVAILABLE}{/datanode}
[junit4] 2> 1220027 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@7c478c1d{HTTP/1.1,[http/1.1]}{localhost:39173}
[junit4] 2> 1220028 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server Started @1220081ms
[junit4] 2> 1220046 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.s.c.MetricsLoggerTask Metrics logging will not be async since the
logger is not log4j
[junit4] 2> 1220098 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 1220099 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1220100 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1220100 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1220100 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1220101 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@4956b200{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,AVAILABLE}
[junit4] 2> 1220268 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x7b6fa862ba7d8158: Processing first storage report for
DS-a82881b2-32d2-45b2-b546-85b122c5cc1b from datanode
b468c6ca-86f7-474e-8a52-590b31ac10c0
[junit4] 2> 1220268 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x7b6fa862ba7d8158: from storage
DS-a82881b2-32d2-45b2-b546-85b122c5cc1b node
DatanodeRegistration(127.0.0.1:39481,
datanodeUuid=b468c6ca-86f7-474e-8a52-590b31ac10c0, infoPort=37426,
infoSecurePort=0, ipcPort=41367,
storageInfo=lv=-57;cid=testClusterID;nsid=60639002;c=1550937508556), blocks: 0,
hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
[junit4] 2> 1220268 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x7b6fa862ba7d8158: Processing first storage report for
DS-b40f295b-4ce9-4bcc-b301-6edec372057c from datanode
b468c6ca-86f7-474e-8a52-590b31ac10c0
[junit4] 2> 1220268 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x7b6fa862ba7d8158: from storage
DS-b40f295b-4ce9-4bcc-b301-6edec372057c node
DatanodeRegistration(127.0.0.1:39481,
datanodeUuid=b468c6ca-86f7-474e-8a52-590b31ac10c0, infoPort=37426,
infoSecurePort=0, ipcPort=41367,
storageInfo=lv=-57;cid=testClusterID;nsid=60639002;c=1550937508556), blocks: 0,
hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
[junit4] 2> 1220322 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.w.WebAppContext@5ce6f113{datanode,/,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/jetty-localhost-34591-datanode-_-any-6381145968145431572.dir/webapp/,AVAILABLE}{/datanode}
[junit4] 2> 1220322 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@68c84b79{HTTP/1.1,[http/1.1]}{localhost:34591}
[junit4] 2> 1220322 INFO
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.Server Started @1220375ms
[junit4] 2> 1220344 WARN
(SUITE-HdfsChaosMonkeyNothingIsSafeTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.s.c.MetricsLoggerTask Metrics logging will not be async since the
logger is not log4j
[junit4] 2> 1220524 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x747d87db2230e0ff: Processing first storage report for
DS-cca828ee-3fbe-4d59-aaa0-8192a8587ccb from datanode
96a74d7d-060a-4eae-bcad-9b4d82475fb1
[junit4] 2> 1220524 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x747d87db2230e0ff: from storage
DS-cca828ee-3fbe-4d59-aaa0-8192a8587ccb node
DatanodeRegistration(127.0.0.1:40794,
datanodeUuid=96a74d7d-060a-4eae-bcad-9b4d82475fb1, infoPort=40756,
infoSecurePort=0, ipcPort=41232,
storageInfo=lv=-57;cid=testClusterID;nsid=60639002;c=1550937508556), blocks: 0,
hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
[junit4] 2> 1220524 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x747d87db2230e0ff: Processing first storage report for
DS-475abb34-a407-4879-8098-5066cb3237e3 from datanode
96a74d7d-060a-4eae-bcad-9b4d82475fb1
[junit4] 2> 1220524 INFO (Block report processor) [ ] BlockStateChange
BLOCK* processReport 0x747d87db2230e0ff: from storage
DS-475abb34-a407-4879-8098-5066cb3237e3 node
DatanodeRegistration(127.0.0.1:40794,
datanodeUuid=96a74d7d-060a-4eae-bcad-9b4d82475fb1, infoPort=40756,
infoSecurePort=0, ipcPort=41232,
storageInfo=lv=-57;cid=testClusterID;nsid=60639002;c=1550937508556), blocks: 0,
hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
[junit4] 2> 1220687 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1220692 INFO (ZkTestServer Run Thread) [ ]
o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1220692 INFO (ZkTestServer Run Thread) [ ]
o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1220792 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer start zk server on port:44434
[junit4] 2> 1220792 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:44434
[junit4] 2> 1220792 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer connecting to 127.0.0.1 44434
[junit4] 2> 1220825 INFO (zkConnectionManagerCallback-7803-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1220838 INFO (zkConnectionManagerCallback-7805-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1220839 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
to /configs/conf1/solrconfig.xml
[junit4] 2> 1220841 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
to /configs/conf1/schema.xml
[junit4] 2> 1220842 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1220843 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
to /configs/conf1/stopwords.txt
[junit4] 2> 1220844 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
to /configs/conf1/protwords.txt
[junit4] 2> 1220845 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
to /configs/conf1/currency.xml
[junit4] 2> 1220846 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
to /configs/conf1/enumsConfig.xml
[junit4] 2> 1220848 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1220849 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1220850 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
to /configs/conf1/old_synonyms.txt
[junit4] 2> 1220851 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkTestServer put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
to /configs/conf1/synonyms.txt
[junit4] 2> 1220852 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly
asked otherwise
[junit4] 2> 1221065 WARN
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 1221065 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 1221065 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 1221065 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1221066 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1221066 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1221066 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1221069 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@116a7fb8{/mz/x,null,AVAILABLE}
[junit4] 2> 1221069 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@2bb26687{HTTP/1.1,[http/1.1,
h2c]}{127.0.0.1:38515}
[junit4] 2> 1221069 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.s.Server Started @1221122ms
[junit4] 2> 1221069 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=hdfs://localhost:38559/hdfs__localhost_38559__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001_tempDir-002_control_data,
hostContext=/mz/x, hostPort=38515,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/control-001/cores}
[junit4] 2> 1221070 ERROR
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 1221070 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter Using logger factory
org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1221070 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
9.0.0
[junit4] 2> 1221070 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1221070 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1221070 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2019-02-23T15:58:30.460Z
[junit4] 2> 1221072 INFO (zkConnectionManagerCallback-7807-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1221073 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1221073 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/control-001/solr.xml
[junit4] 2> 1221078 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 1221078 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1221080 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1221164 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized:
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 1221166 WARN
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@25acbd01[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1221171 WARN
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@69f9ab58[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1221172 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44434/solr
[junit4] 2> 1221174 INFO (zkConnectionManagerCallback-7813-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1221176 INFO (zkConnectionManagerCallback-7815-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1221304 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.OverseerElectionContext I am going to be
the leader 127.0.0.1:38515_mz%2Fx
[junit4] 2> 1221305 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.Overseer Overseer
(id=72848762358202372-127.0.0.1:38515_mz%2Fx-n_0000000000) starting
[junit4] 2> 1221319 INFO (zkConnectionManagerCallback-7822-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1221323 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster
at 127.0.0.1:44434/solr ready
[junit4] 2> 1221324 INFO
(OverseerStateUpdate-72848762358202372-127.0.0.1:38515_mz%2Fx-n_0000000000)
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.Overseer Starting to work on the main
queue : 127.0.0.1:38515_mz%2Fx
[junit4] 2> 1221325 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:38515_mz%2Fx
[junit4] 2> 1221333 INFO (zkCallback-7814-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1221345 INFO (zkCallback-7821-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1221350 DEBUG
(OverseerAutoScalingTriggerThread-72848762358202372-127.0.0.1:38515_mz%2Fx-n_0000000000)
[ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial
livenodes: [127.0.0.1:38515_mz%2Fx]
[junit4] 2> 1221352 DEBUG (ScheduledTrigger-2980-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1221354 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.h.a.MetricsHistoryHandler No .system
collection, keeping metrics history in memory.
[junit4] 2> 1221381 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1221395 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1221396 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1221397 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94])
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/control-001/cores
[junit4] 2> 1221434 INFO (zkConnectionManagerCallback-7828-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1221435 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1221437 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44434/solr ready
[junit4] 2> 1221438 INFO (qtp787767952-16090) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38515_mz%252Fx&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1221443 INFO
(OverseerThreadFactory-2982-thread-1-processing-n:127.0.0.1:38515_mz%2Fx)
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.a.c.CreateCollectionCmd Create
collection control_collection
[junit4] 2> 1221567 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core
create command
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1221568 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault
Allocating transient cache for 4 transient cores
[junit4] 2> 1222354 DEBUG (ScheduledTrigger-2980-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1222596 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
[junit4] 2> 1222612 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 1222724 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1222748 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1'
using configuration from collection control_collection, trusted=true
[junit4] 2> 1222748 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.control_collection.shard1.replica_n1' (registry
'solr.core.control_collection.shard1.replica_n1') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1222748 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory
solr.hdfs.home=hdfs://localhost:38559/solr_hdfs_home
[junit4] 2> 1222748 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 1222748 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore
at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/control-001/cores/control_collection_shard1_replica_n1],
dataDir=[hdfs://localhost:38559/solr_hdfs_home/control_collection/core_node2/data/]
[junit4] 2> 1222750 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:38559/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
[junit4] 2> 1222760 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 1222760 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 1222760 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
[junit4] 2> 1222778 WARN (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds
and try again.
[junit4] 2> 1223355 DEBUG (ScheduledTrigger-2980-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1224355 DEBUG (ScheduledTrigger-2980-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1225355 DEBUG (ScheduledTrigger-2980-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1226355 DEBUG (ScheduledTrigger-2980-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1227355 DEBUG (ScheduledTrigger-2980-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1227779 WARN (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds
and try again.
[junit4] 2> 1228355 DEBUG (ScheduledTrigger-2980-thread-1) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1229356 DEBUG (ScheduledTrigger-2980-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1230356 DEBUG (ScheduledTrigger-2980-thread-2) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1231356 DEBUG (ScheduledTrigger-2980-thread-4) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1232356 DEBUG (ScheduledTrigger-2980-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1232964 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 1232966 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:38559/solr_hdfs_home/control_collection/core_node2/data
[junit4] 2> 1233019 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:38559/solr_hdfs_home/control_collection/core_node2/data/index
[junit4] 2> 1233036 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 1233036 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 1233040 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 1233041 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.MockRandomMergePolicy:
org.apache.lucene.index.MockRandomMergePolicy@b558f02
[junit4] 2> 1233240 WARN (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 1233356 DEBUG (ScheduledTrigger-2980-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1233411 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 1233411 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1233411 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 1233425 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
[junit4] 2> 1233425 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1233436 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=22, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=50.7861328125,
floorSegmentMB=0.9345703125, forceMergeDeletesPctAllowed=25.986590869294186,
segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0,
deletesPctAllowed=30.38741289016026
[junit4] 2> 1233476 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@23aaef91[control_collection_shard1_replica_n1] main]
[junit4] 2> 1233478 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 1233478 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1233479 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1233504 WARN (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality information for
hdfs://localhost:43337/solr3 due to exception: java.net.ConnectException: Call
From lucene1-us-west/10.41.0.5 to localhost:43337 failed on connection
exception: java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
[junit4] 2> 1233504 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using
new clock 1626275864006098944
[junit4] 2> 1233516 INFO
(searcherExecutor-2987-thread-1-processing-n:127.0.0.1:38515_mz%2Fx
x:control_collection_shard1_replica_n1 c:control_collection s:shard1)
[n:127.0.0.1:38515_mz%2Fx c:control_collection s:shard1
x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore
[control_collection_shard1_replica_n1] Registered new searcher
Searcher@23aaef91[control_collection_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1233521 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ZkShardTerms Successful update of terms at
/collections/control_collection/terms/shard1 to Terms{values={core_node2=0},
version=0}
[junit4] 2> 1233521 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContextBase make sure parent is created
/collections/control_collection/leaders/shard1
[junit4] 2> 1233537 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1233537 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1233537 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy Sync replicas to
http://127.0.0.1:38515/mz/x/control_collection_shard1_replica_n1/
[junit4] 2> 1233538 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1233538 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.SyncStrategy
http://127.0.0.1:38515/mz/x/control_collection_shard1_replica_n1/ has no
replicas
[junit4] 2> 1233538 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
/collections/control_collection/leaders/shard1/leader after winning as
/collections/control_collection/leader_elect/shard1/election/72848762358202372-core_node2-n_0000000000
[junit4] 2> 1233542 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:38515/mz/x/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 1233544 INFO (zkCallback-7814-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1233545 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1233547 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
c:control_collection s:shard1 x:control_collection_shard1_replica_n1]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=11979
[junit4] 2> 1233549 INFO (zkCallback-7814-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1233551 INFO (qtp787767952-16090) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at
most 45 seconds. Check all shard replicas
[junit4] 2> 1233650 INFO (zkCallback-7814-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1233650 INFO (zkCallback-7814-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1233651 INFO (qtp787767952-16090) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38515_mz%252Fx&wt=javabin&version=2}
status=0 QTime=12213
[junit4] 2> 1233652 INFO (zkCallback-7814-thread-3) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1233681 INFO (zkConnectionManagerCallback-7834-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1233682 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1233683 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44434/solr ready
[junit4] 2> 1233683 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection
loss:false
[junit4] 2> 1233684 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1233695 INFO
(OverseerThreadFactory-2982-thread-2-processing-n:127.0.0.1:38515_mz%2Fx)
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.a.c.CreateCollectionCmd Create
collection collection1
[junit4] 2> 1233695 INFO
(OverseerCollectionConfigSetProcessor-72848762358202372-127.0.0.1:38515_mz%2Fx-n_0000000000)
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1233902 WARN
(OverseerThreadFactory-2982-thread-2-processing-n:127.0.0.1:38515_mz%2Fx)
[n:127.0.0.1:38515_mz%2Fx ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
create a collection (collection1) without cores.
[junit4] 2> 1233904 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at
most 45 seconds. Check all shard replicas
[junit4] 2> 1233905 INFO (qtp787767952-16092) [n:127.0.0.1:38515_mz%2Fx
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2}
status=0 QTime=220
[junit4] 2> 1233906 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances
pullReplicaCount=0 numOtherReplicas=5
[junit4] 2> 1234075 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-1-001
of type NRT
[junit4] 2> 1234090 WARN (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 1234091 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 1234091 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 1234091 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1234119 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1234119 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1234119 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1234119 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@74f1dd51{/mz/x,null,AVAILABLE}
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@459354e2{HTTP/1.1,[http/1.1,
h2c]}{127.0.0.1:41941}
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.e.j.s.Server Started @1234173ms
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=hdfs://localhost:38559/hdfs__localhost_38559__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001_tempDir-002_jetty1,
replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/mz/x, hostPort=41941,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-1-001/cores}
[junit4] 2> 1234120 ERROR (closeThreadPool-7835-thread-1) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter Using logger factory
org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
9.0.0
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1234120 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2019-02-23T15:58:43.510Z
[junit4] 2> 1234123 INFO (zkConnectionManagerCallback-7837-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234124 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1234124 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-1-001/solr.xml
[junit4] 2> 1234128 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 1234128 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1234131 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1234331 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-2-001
of type NRT
[junit4] 2> 1234351 WARN (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 1234351 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 1234351 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 1234351 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1234357 DEBUG (ScheduledTrigger-2980-thread-3) [ ]
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with
currently live nodes: 1 and last live nodes: 1
[junit4] 2> 1234380 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized:
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 1234381 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1234381 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1234381 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1234381 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@581c0de8{/mz/x,null,AVAILABLE}
[junit4] 2> 1234382 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@4d585c04{HTTP/1.1,[http/1.1,
h2c]}{127.0.0.1:43852}
[junit4] 2> 1234382 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.Server Started @1234435ms
[junit4] 2> 1234382 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=hdfs://localhost:38559/hdfs__localhost_38559__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001_tempDir-002_jetty2,
replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/mz/x, hostPort=43852,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-2-001/cores}
[junit4] 2> 1234382 ERROR (closeThreadPool-7835-thread-2) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 1234382 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter Using logger factory
org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1234383 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
9.0.0
[junit4] 2> 1234383 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1234383 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1234383 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2019-02-23T15:58:43.773Z
[junit4] 2> 1234385 WARN (closeThreadPool-7835-thread-1) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@746a1ec[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234396 INFO (zkConnectionManagerCallback-7840-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234407 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1234407 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-2-001/solr.xml
[junit4] 2> 1234409 WARN (closeThreadPool-7835-thread-1) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@795228d1[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234410 INFO (closeThreadPool-7835-thread-1) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44434/solr
[junit4] 2> 1234416 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 1234416 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1234418 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1234435 INFO (zkConnectionManagerCallback-7845-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234458 INFO (zkConnectionManagerCallback-7847-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234464 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (1)
[junit4] 2> 1234468 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.ZkController Publish
node=127.0.0.1:41941_mz%2Fx as DOWN
[junit4] 2> 1234470 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.TransientSolrCoreCacheDefault Allocating
transient cache for 4 transient cores
[junit4] 2> 1234470 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:41941_mz%2Fx
[junit4] 2> 1234471 INFO (zkCallback-7821-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1234472 INFO (zkCallback-7833-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1234472 INFO (zkCallback-7814-thread-3) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1234478 INFO (zkCallback-7846-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1234528 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized:
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 1234539 WARN (closeThreadPool-7835-thread-2) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@8fd0177[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234542 INFO (zkConnectionManagerCallback-7854-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234543 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (2)
[junit4] 2> 1234544 WARN (closeThreadPool-7835-thread-2) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@74005405[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234545 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster
at 127.0.0.1:44434/solr ready
[junit4] 2> 1234545 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44434/solr
[junit4] 2> 1234546 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.h.a.MetricsHistoryHandler No .system
collection, keeping metrics history in memory.
[junit4] 2> 1234553 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-3-001
of type NRT
[junit4] 2> 1234556 INFO (zkConnectionManagerCallback-7861-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234563 WARN (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 1234571 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 1234571 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 1234572 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1234579 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234595 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234595 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234597 INFO (closeThreadPool-7835-thread-1)
[n:127.0.0.1:41941_mz%2Fx ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-1-001/cores
[junit4] 2> 1234606 INFO (zkConnectionManagerCallback-7863-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234611 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (2)
[junit4] 2> 1234616 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.ZkController Publish
node=127.0.0.1:43852_mz%2Fx as DOWN
[junit4] 2> 1234624 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.TransientSolrCoreCacheDefault Allocating
transient cache for 4 transient cores
[junit4] 2> 1234624 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:43852_mz%2Fx
[junit4] 2> 1234626 INFO (zkCallback-7814-thread-3) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234626 INFO (zkCallback-7821-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234626 INFO (zkCallback-7833-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234627 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1234627 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1234627 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1234627 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@3fa4de0c{/mz/x,null,AVAILABLE}
[junit4] 2> 1234633 INFO (zkCallback-7846-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234634 INFO (zkCallback-7862-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234634 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@7958e2b6{HTTP/1.1,[http/1.1,
h2c]}{127.0.0.1:44073}
[junit4] 2> 1234634 INFO (closeThreadPool-7835-thread-3) [ ]
o.e.j.s.Server Started @1234687ms
[junit4] 2> 1234634 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=hdfs://localhost:38559/hdfs__localhost_38559__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001_tempDir-002_jetty3,
replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/mz/x, hostPort=44073,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-3-001/cores}
[junit4] 2> 1234634 ERROR (closeThreadPool-7835-thread-3) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 1234634 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter Using logger factory
org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1234634 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
9.0.0
[junit4] 2> 1234635 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1234635 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1234635 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2019-02-23T15:58:44.025Z
[junit4] 2> 1234635 INFO (zkCallback-7853-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1234637 INFO (zkConnectionManagerCallback-7867-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234638 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1234638 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-3-001/solr.xml
[junit4] 2> 1234645 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 1234645 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1234647 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1234674 INFO (zkConnectionManagerCallback-7872-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234676 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (3)
[junit4] 2> 1234678 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster
at 127.0.0.1:44434/solr ready
[junit4] 2> 1234680 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.h.a.MetricsHistoryHandler No .system
collection, keeping metrics history in memory.
[junit4] 2> 1234709 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234727 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234727 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1234729 INFO (closeThreadPool-7835-thread-2)
[n:127.0.0.1:43852_mz%2Fx ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-2-001/cores
[junit4] 2> 1234793 INFO
(TEST-HdfsChaosMonkeyNothingIsSafeTest.test-seed#[29E266F0FBEBE94]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-4-001
of type NRT
[junit4] 2> 1234793 WARN (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 1234794 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 1234794 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 1234794 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git:
c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 1234834 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1234834 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1234834 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1234835 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@295ed2a2{/mz/x,null,AVAILABLE}
[junit4] 2> 1234835 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@1c30732c{HTTP/1.1,[http/1.1,
h2c]}{127.0.0.1:38233}
[junit4] 2> 1234835 INFO (closeThreadPool-7835-thread-2) [ ]
o.e.j.s.Server Started @1234888ms
[junit4] 2> 1234835 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=hdfs://localhost:38559/hdfs__localhost_38559__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-BadApples-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001_tempDir-002_jetty4,
replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/mz/x, hostPort=38233,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-4-001/cores}
[junit4] 2> 1234836 ERROR (closeThreadPool-7835-thread-2) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 1234836 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter Using logger factory
org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1234836 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
9.0.0
[junit4] 2> 1234836 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1234836 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1234836 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2019-02-23T15:58:44.226Z
[junit4] 2> 1234845 INFO (zkConnectionManagerCallback-7875-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234850 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1234850 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest_29E266F0FBEBE94-001/shard-4-001/solr.xml
[junit4] 2> 1234855 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay
is ignored
[junit4] 2> 1234855 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1234857 INFO (closeThreadPool-7835-thread-2) [ ]
o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1234870 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized:
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 1234872 WARN (closeThreadPool-7835-thread-3) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@15eafbda[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234876 WARN (closeThreadPool-7835-thread-3) [ ]
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for
SslContextFactory@777836a6[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 1234877 INFO (closeThreadPool-7835-thread-3) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44434/solr
[junit4] 2> 1234879 INFO (zkConnectionManagerCallback-7881-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234881 INFO (zkConnectionManagerCallback-7883-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234886 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (3)
[junit4] 2> 1234890 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.ZkController Publish
node=127.0.0.1:44073_mz%2Fx as DOWN
[junit4] 2> 1234892 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.TransientSolrCoreCacheDefault Allocating
transient cache for 4 transient cores
[junit4] 2> 1234892 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:44073_mz%2Fx
[junit4] 2> 1234894 INFO (zkCallback-7846-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234894 INFO (zkCallback-7833-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234894 INFO (zkCallback-7821-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234894 INFO (zkCallback-7814-thread-3) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234904 INFO (zkCallback-7853-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234904 INFO (zkCallback-7871-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234904 INFO (zkCallback-7862-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234905 INFO (zkCallback-7882-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1234966 INFO (zkConnectionManagerCallback-7890-thread-1) [
] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1234967 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (4)
[junit4] 2> 1234969 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster
at 127.0.0.1:44434/solr ready
[junit4] 2> 1234973 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.h.a.MetricsHistoryHandler No .system
collection, keeping metrics history in memory.
[junit4] 2> 1235002 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.node' (registry 'solr.node') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1235017 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jvm' (registry 'solr.jvm') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1235017 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.jetty' (registry 'solr.jetty') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3adb243e
[junit4] 2> 1235019 INFO (closeThreadPool-7835-thread-3)
[n:127.0.0.1:44073_mz%2Fx ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/sol
[...truncated too long message...]
ava:748)
[junit4] 2> 2325393 INFO
(TEST-StressHdfsTest.test-seed#[29E266F0FBEBE94]) [ ] o.a.s.c.ZkTestServer
parse host and port list: 127.0.0.1:38111
[junit4] 2> 2325393 INFO
(TEST-StressHdfsTest.test-seed#[29E266F0FBEBE94]) [ ] o.a.s.c.ZkTestServer
connecting to 127.0.0.1 38111
[junit4] 2> 2325400 WARN
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 2325554 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.w.WebAppContext@78dd6717{datanode,/,null,UNAVAILABLE}{/datanode}
[junit4] 2> 2325554 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Stopped
ServerConnector@2746a3e3{HTTP/1.1,[http/1.1]}{localhost:0}
[junit4] 2> 2325555 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ] o.e.j.s.session
node0 Stopped scavenging
[junit4] 2> 2325555 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@2825f12e{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE}
[junit4] 2> 2325558 WARN (BP-110225281-10.41.0.5-1550938479821
heartbeating to localhost/127.0.0.1:41833) [ ]
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager
interrupted
[junit4] 2> 2325558 WARN (BP-110225281-10.41.0.5-1550938479821
heartbeating to localhost/127.0.0.1:41833) [ ] o.a.h.h.s.d.DataNode Ending
block pool service for: Block pool BP-110225281-10.41.0.5-1550938479821
(Datanode Uuid fa805903-c767-4f13-969d-8d8ea44025e8) service to
localhost/127.0.0.1:41833
[junit4] 2> 2325576 WARN
(refreshUsed-/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_29E266F0FBEBE94-001/tempDir-001/hdfsBaseDir/data/data3/current/BP-110225281-10.41.0.5-1550938479821)
[ ] o.a.h.f.CachingGetSpaceUsed Thread Interrupted waiting to refresh disk
information: sleep interrupted
[junit4] 2> 2325637 WARN
(refreshUsed-/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_29E266F0FBEBE94-001/tempDir-001/hdfsBaseDir/data/data4/current/BP-110225281-10.41.0.5-1550938479821)
[ ] o.a.h.f.CachingGetSpaceUsed Thread Interrupted waiting to refresh disk
information: sleep interrupted
[junit4] 2> 2325646 WARN
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
[junit4] 2> 2325786 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.w.WebAppContext@47bd7c13{datanode,/,null,UNAVAILABLE}{/datanode}
[junit4] 2> 2325787 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Stopped
ServerConnector@18f8378f{HTTP/1.1,[http/1.1]}{localhost:0}
[junit4] 2> 2325787 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ] o.e.j.s.session
node0 Stopped scavenging
[junit4] 2> 2325787 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@35ec66f4{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE}
[junit4] 2> 2325789 WARN (BP-110225281-10.41.0.5-1550938479821
heartbeating to localhost/127.0.0.1:41833) [ ]
o.a.h.h.s.d.IncrementalBlockReportManager IncrementalBlockReportManager
interrupted
[junit4] 2> 2325789 WARN (BP-110225281-10.41.0.5-1550938479821
heartbeating to localhost/127.0.0.1:41833) [ ] o.a.h.h.s.d.DataNode Ending
block pool service for: Block pool BP-110225281-10.41.0.5-1550938479821
(Datanode Uuid 42897d9a-7567-49fb-b3f6-28484defb9b6) service to
localhost/127.0.0.1:41833
[junit4] 2> 2325802 WARN
(refreshUsed-/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_29E266F0FBEBE94-001/tempDir-001/hdfsBaseDir/data/data1/current/BP-110225281-10.41.0.5-1550938479821)
[ ] o.a.h.f.CachingGetSpaceUsed Thread Interrupted waiting to refresh disk
information: sleep interrupted
[junit4] 2> 2325830 WARN
(refreshUsed-/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_29E266F0FBEBE94-001/tempDir-001/hdfsBaseDir/data/data2/current/BP-110225281-10.41.0.5-1550938479821)
[ ] o.a.h.f.CachingGetSpaceUsed Thread Interrupted waiting to refresh disk
information: sleep interrupted
[junit4] 2> 2325864 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.w.WebAppContext@358afb7{hdfs,/,null,UNAVAILABLE}{/hdfs}
[junit4] 2> 2325869 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.AbstractConnector Stopped
ServerConnector@5dff9c94{HTTP/1.1,[http/1.1]}{localhost:0}
[junit4] 2> 2325869 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ] o.e.j.s.session
node0 Stopped scavenging
[junit4] 2> 2325870 INFO
(SUITE-StressHdfsTest-seed#[29E266F0FBEBE94]-worker) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@32343883{static,/static,jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-3.2.0-tests.jar!/webapps/static,UNAVAILABLE}
[junit4] 2> NOTE: leaving temporary files on disk at:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_29E266F0FBEBE94-001
[junit4] 2> Feb 23, 2019 4:16:56 PM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 32 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@be7b5fb),
locale=zh, timezone=America/Cambridge_Bay
[junit4] 2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation
1.8.0_191 (64-bit)/cpus=4,threads=4,free=133881712,total=473956352
[junit4] 2> NOTE: All tests run in this JVM: [JvmMetricsTest,
TestScoreJoinQPNoScore, BasicFunctionalityTest, MultiSolrCloudTestCaseTest,
ExplicitHLLTest, DistributedUpdateProcessorTest, TestDocTermOrds,
TestTolerantSearch, TestHighlightDedupGrouping, ZkSolrClientTest,
TestCloudJSONFacetSKG, TestPartialUpdateDeduplication, TestSimpleQParserPlugin,
TestCursorMarkWithoutUniqueKey, SchemaVersionSpecificBehaviorTest,
CdcrWithNodesRestartsTest, RequestHandlersTest, TestRestoreCore,
IndexSizeTriggerTest, TestSolrXml, TestPayloadScoreQParserPlugin,
FullSolrCloudDistribCmdsTest, SolrJmxReporterTest,
PreAnalyzedUpdateProcessorTest, OverseerModifyCollectionTest,
ExitableDirectoryReaderTest, TestDynamicFieldResource,
TlogReplayBufferedWhileIndexingTest, URLClassifyProcessorTest,
LoggingHandlerTest, UtilsToolTest, CollectionPropsTest, RankQueryTest,
MetricsHistoryIntegrationTest, TestNoOpRegenerator,
RecoveryAfterSoftCommitTest, MoveReplicaHDFSFailoverTest, JsonLoaderTest,
TestUtilizeNode, SuggesterTest, TestCorePropertiesReload, ZkStateWriterTest,
HLLUtilTest, CloudExitableDirectoryReaderTest, TestFoldingMultitermQuery,
TestSimPolicyCloud, TestRandomFaceting, TestNumericTerms32, UUIDFieldTest,
SimpleFacetsTest, TestReloadDeadlock, TestFieldCacheVsDocValues, RootFieldTest,
PeerSyncWithIndexFingerprintCachingTest, TestCollapseQParserPlugin,
SolrGangliaReporterTest, TestSimExecutePlanAction, TestCloudManagedSchema,
JavabinLoaderTest, OutOfBoxZkACLAndCredentialsProvidersTest,
TestComponentsName, TestFaceting, HdfsBasicDistributedZkTest,
CreateCollectionCleanupTest, SortSpecParsingTest,
DistributedFacetPivotWhiteBoxTest, DistributedFacetPivotLongTailTest,
TestPerFieldSimilarityWithDefaultOverride, TestEmbeddedSolrServerConstructors,
BufferStoreTest, DistribDocExpirationUpdateProcessorTest,
DeleteLastCustomShardedReplicaTest, TestPointFields, TestSimLargeCluster,
ProtectedTermFilterFactoryTest, PrimitiveFieldTypeTest, SolrShardReporterTest,
TestSolr4Spatial2, BlockJoinFacetDistribTest, SolrInfoBeanTest,
LeaderElectionTest, ConcurrentDeleteAndCreateCollectionTest,
TestCloudRecovery2, TestLegacyBM25SimilarityFactory, CopyFieldTest,
TestCryptoKeys, StressHdfsTest]
[junit4] Completed [351/844 (2!)] on J1 in 138.30s, 1 test, 1 error <<<
FAILURES!
[...truncated 1632 lines...]
[junit4] JVM J1: stdout was not empty, see:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20190223_153809_0964861013134496910624.sysout
[junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/heapdumps/java_pid15365.hprof
...
[junit4] Heap dump file created [616841767 bytes in 41.047 secs]
[junit4] <<< JVM J1: EOF ----
[...truncated 8929 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:661:
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:585:
Some of the tests produced a heap dump, but did not fail. Maybe a suppressed
OutOfMemoryError? Dumps created:
* java_pid15365.hprof
Total time: 298 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
