Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2267/

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([92DAA5CA5D099EAA:8A602DC6533C5345]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:862)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
        at 
__randomizedtesting.SeedInfo.seed([92DAA5CA5D099EAA:CF92BB4392CF38E5]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1173)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12082 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_92DAA5CA5D099EAA-001/init-core-data-001
   [junit4]   2> 1057230 WARN  
(SUITE-TestTriggerIntegration-seed#[92DAA5CA5D099EAA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1057230 INFO  
(SUITE-TestTriggerIntegration-seed#[92DAA5CA5D099EAA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1057232 INFO  
(SUITE-TestTriggerIntegration-seed#[92DAA5CA5D099EAA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1057232 INFO  
(SUITE-TestTriggerIntegration-seed#[92DAA5CA5D099EAA]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1057656 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
   [junit4]   2> 1057657 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1057657 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1057708 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1057708 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 1057708 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10000_solr]
   [junit4]   2> 1057708 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1057709 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1057744 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1057744 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1057760 DEBUG (ScheduledTrigger-1712-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1057805 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
   [junit4]   2> 1057805 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1057821 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2>        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1057830 DEBUG (ScheduledTrigger-1712-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1057927 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1057935 WARN  (ScheduledTrigger-1712-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger ran but was already closed
   [junit4]   2> 1057971 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 1
   [junit4]   2> 1057971 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1057971 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1057971 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1057972 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10000_solr
   [junit4]   2> 1057991 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1057991 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1058008 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058008 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
   [junit4]   2> 1058044 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058072 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
   [junit4]   2> 1058186 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058271 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058289 DEBUG (simCloudManagerPool-1714-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1058290 DEBUG (simCloudManagerPool-1714-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
2
   [junit4]   2> 1058324 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058324 DEBUG (simCloudManagerPool-1714-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1058326 DEBUG (simCloudManagerPool-1714-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1058326 DEBUG (simCloudManagerPool-1714-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1058326 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 1058344 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058326 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1058345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 1058344 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1058345 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
3
   [junit4]   2> 1058345 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1058345 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1058345 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10001_solr, 
127.0.0.1:10002_solr]
   [junit4]   2> 1058348 DEBUG (simCloudManagerPool-1714-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1058348 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
   [junit4]   2> 1058356 DEBUG (ScheduledTrigger-1715-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1058356 DEBUG (ScheduledTrigger-1715-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1058356 DEBUG (ScheduledTrigger-1715-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 
973799745554613
   [junit4]   2> 1058360 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1058360 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1058360 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1058360 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3
   [junit4]   2> 1058372 DEBUG (ScheduledTrigger-1715-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1058397 DEBUG (ScheduledTrigger-1715-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1058397 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1058397 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1058397 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10003_solr] added at times 
[973799745554613], now=973801766859363
   [junit4]   2> 1058397 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973799745554613,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973799745554613],
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1058398 DEBUG (ScheduledTrigger-1715-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_trigger, node_added_trigger]
   [junit4]   2> 1058399 DEBUG (ScheduledTrigger-1715-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 3
   [junit4]   2> 1058492 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 2
   [junit4]   2> 1058492 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1058492 DEBUG 
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1058492 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1058460 DEBUG (AutoscalingActionExecutor-1716-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973799745554613,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973799745554613],
   [junit4]   2>     "_enqueue_time_":973801824403713,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"]}}
   [junit4]   2> 1058495 WARN  (AutoscalingActionExecutor-1716-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTrigger node_added_trigger has been closed.
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:525)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:309)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1058497 DEBUG (AutoscalingActionExecutor-1716-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 1849 ms for event 
id=375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS 
############
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:         2
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:        2
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:         2
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##             127.0.0.1:10000_solr
   [junit4]   2> 1058501 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##             127.0.0.1:10001_solr
   [junit4]   2> 1058595 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 3
   [junit4]   2> 1058608 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:        []
   [junit4]   2> 1058608 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:      0
   [junit4]   2> 1058608 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:      0
   [junit4]   2> 1058608 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:             0
   [junit4]   2> 1058608 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts 
##########
   [junit4]   2> 1058609 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##             - autoscaling        2
   [junit4]   2> 1058609 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##             - update             1
   [junit4]   2> 1058609 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 1058610 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
   [junit4]   2> 1058610 INFO  
(TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##             - STARTED         1
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}],
 enabled=true}
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion -1
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 1058644 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1058644 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 
127.0.0.1:10001_solr
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration -Dtests.method=testNodeMarkersRegistration 
-Dtests.seed=92DAA5CA5D099EAA -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=bg -Dtests.timezone=Asia/Phnom_Penh -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.25s J2 | 
TestTriggerIntegration.testNodeMarkersRegistration <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([92DAA5CA5D099EAA:8A602DC6533C5345]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:862)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1058664 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
   [junit4]   2> 1058664 DEBUG (ScheduledTrigger-1718-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with 
currently live nodes: 2
   [junit4]   2> 1058664 DEBUG (ScheduledTrigger-1718-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
   [junit4]   2> 1058676 DEBUG (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973799745554613,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973799745554613],
   [junit4]   2>     "_dequeue_time_":973815732508763,
   [junit4]   2>     "_enqueue_time_":973801824403713,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1058680 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 1058712 DEBUG (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_trigger, node_added_trigger]
   [junit4]   2> 1058717 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
   [junit4]   2> 1058717 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
4
   [junit4]   2> 1058717 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10001_solr
   [junit4]   2> 1058717 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 
127.0.0.1:10000_solr
   [junit4]   2> 1058717 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 1058717 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1058729 WARN  (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:460)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:122)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:744)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:714)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$4(ScheduledTriggers.java:274)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.run(ScheduledTriggers.java:553)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:191)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:458)
   [junit4]   2>        ... 12 more
   [junit4]   2> 1058729 WARN  (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Ignoring autoscaling event {
   [junit4]   2>   "id":"375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973799745554613,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973799745554613],
   [junit4]   2>     "_dequeue_time_":973815732508763,
   [junit4]   2>     "_enqueue_time_":973801824403713,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"],
   [junit4]   2>     "replaying":true}} from trigger 
org.apache.solr.cloud.autoscaling.NodeAddedTrigger@2570254f because the 
executor has already been closed
   [junit4]   2> 1058730 ERROR (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Failed to re-play event, discarding: {
   [junit4]   2>   "id":"375aa6bea88b5Te19v0m1w0ylx6923iq76qlalw",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973799745554613,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973799745554613],
   [junit4]   2>     "_dequeue_time_":973815732508763,
   [junit4]   2>     "_enqueue_time_":973801824403713,
   [junit4]   2>     "nodeNames":["127.0.0.1:10003_solr"],
   [junit4]   2>     "replaying":true}}
   [junit4]   2> 1058730 WARN  (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger ran but was already closed
   [junit4]   2> 1058730 ERROR (ScheduledTrigger-1718-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Unexpected exception in NodeAddedTrigger
   [junit4]   2> java.lang.RuntimeException: Trigger has been closed
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.NodeAddedTrigger.run(NodeAddedTrigger.java:124)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.run(ScheduledTriggers.java:568)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]   2>        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1058737 DEBUG 
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1058754 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1058754 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 5
   [junit4]   2> 1058754 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1058754 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion -1
   [junit4]   2> 1058754 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 1058773 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1058773 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 1058773 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058810 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058956 DEBUG (ScheduledTrigger-1721-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1058988 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059009 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059029 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059061 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059083 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059118 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059191 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059193 DEBUG (simCloudManagerPool-1720-thread-4) [    ] 
o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for collection1 / 
shard1: {"core_node1":{
   [junit4]   2>     "node_name":"127.0.0.1:10002_solr",
   [junit4]   2>     "core":"collection1_shard1_replica_n1",
   [junit4]   2>     "leader":"true",
   [junit4]   2>     "state":"active",
   [junit4]   2>     "type":"NRT",
   [junit4]   2>     "INDEX.sizeInBytes":123450000,
   [junit4]   2>     "shard":"shard1",
   [junit4]   2>     "collection":"collection1"}}
   [junit4]   2> 1059208 DEBUG (simCloudManagerPool-1720-thread-5) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1059208 DEBUG (simCloudManagerPool-1720-thread-5) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
6
   [junit4]   2> 1059227 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059243 DEBUG (simCloudManagerPool-1720-thread-5) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1059247 DEBUG (ScheduledTrigger-1721-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 1059264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1059264 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 1059276 DEBUG (simCloudManagerPool-1720-thread-6) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1059276 DEBUG (simCloudManagerPool-1720-thread-6) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
7
   [junit4]   2> 1059276 DEBUG (simCloudManagerPool-1720-thread-6) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1059276 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 1059277 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1059277 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 1059281 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059302 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059336 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059535 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059557 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059592 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059628 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1059666 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375b53621d01bTe19v0m1w0ylx6923iq76qlam0",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":973846087847963,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10002_solr":250.0,
   [junit4]   2>       "127.0.0.1:10003_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    \"leader\":\"true\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10003_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}}}}
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 
   [junit4]   2> 1059687 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:              2
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:     0
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:              0
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:     [collection1]
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:   1
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:   1
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:          2
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## * collection1            2
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##          - active           2
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##          - CREATE             1
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##          - autoscaling        2
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##          - ABORTED         1
   [junit4]   2> 1059688 INFO  
(TEST-TestTriggerIntegration.testSearchRate-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##          - STARTED         1
   [junit4]   2> 1059701 DEBUG (ScheduledTrigger-1721-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
search_rate_trigger]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestTriggerIntegration -Dtests.method=testSearchRate 
-Dtests.seed=92DAA5CA5D099EAA -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=bg -Dtests.timezone=Asia/Phnom_Penh -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.08s J2 | TestTriggerIntegration.testSearchRate <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The trigger did not 
fire at all
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([92DAA5CA5D099EAA:CF92BB4392CF38E5]:0)
   [junit4]    >        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1173)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1059922 DEBUG (AutoscalingActionExecutor-1722-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375b53621d01bTe19v0m1w0ylx6923iq76qlam0",
   [junit4]   2>   "source":"search_rate_trigger",
   [junit4]   2>   "eventTime":973846087847963,
   [junit4]   2>   "eventType":"SEARCHRATE",
   [junit4]   2>   "properties":{
   [junit4]   2>     "node":{
   [junit4]   2>       "127.0.0.1:10002_solr":250.0,
   [junit4]   2>       "127.0.0.1:10003_solr":250.0},
   [junit4]   2>     "replica":[
   [junit4]   2>       "{\"core_node1\":{\n    
\"core\":\"collection1_shard1_replica_n1\",\n    \"leader\":\"true\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10002_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}",
   [junit4]   2>       "{\"core_node2\":{\n    
\"core\":\"collection1_shard1_replica_n2\",\n    
\"INDEX.sizeInBytes\":123450000,\n    \"rate\":250.0,\n    
\"node_name\":\"127.0.0.1:10003_solr\",\n    \"state\":\"active\",\n    
\"type\":\"NRT\",\n    \"QUERY./select.requestTimes:1minRate\":250.0,\n    
\"shard\":\"shard1\",\n    \"collection\":\"collection1\"}}"],
   [junit4]   2>     "collection":{"collection1":500.0},
   [junit4]   2>     "shard":{"collection1":{"shard1":500.0}},
   [junit4]   2>     "_enqueue_time_":973866981091813}}
   [junit4]   2> 1060049 DEBUG (AutoscalingActionExecutor-1722-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 1060050 DEBUG (AutoscalingActionExecutor-1722-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: search_rate_trigger after 100ms
   [junit4]   2> 1060050 DEBUG (AutoscalingActionExecutor-1722-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 6382 ms for event 
id=375b53621d01bTe19v0m1w0ylx6923iq76qlam0
   [junit4]   2> 1060085 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testListeners
   [junit4]   2> 1060085 DEBUG 
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
8
   [junit4]   2> 1060085 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 1060085 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1060086 DEBUG 
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1060208 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1060208 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 9
   [junit4]   2> 1060208 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060208 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
   [junit4]   2> 1060208 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 1060228 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1060228 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 1060228 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060264 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060407 DEBUG (simCloudManagerPool-1723-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1060407 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060408 DEBUG (simCloudManagerPool-1723-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
10
   [junit4]   2> 1060428 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060451 DEBUG (ScheduledTrigger-1724-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060451 DEBUG (simCloudManagerPool-1723-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060451 DEBUG (simCloudManagerPool-1723-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060451 DEBUG (simCloudManagerPool-1723-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1060463 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
10
   [junit4]   2> 1060463 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1060463 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
   [junit4]   2> 1060463 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1060463 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1060468 DEBUG (simCloudManagerPool-1723-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1060469 DEBUG (simCloudManagerPool-1723-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
11
   [junit4]   2> 1060469 DEBUG (simCloudManagerPool-1723-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060469 DEBUG (simCloudManagerPool-1723-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060469 DEBUG (simCloudManagerPool-1723-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1060479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
11
   [junit4]   2> 1060479 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
   [junit4]   2> 1060479 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060489 DEBUG (simCloudManagerPool-1723-thread-3) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1060489 DEBUG (simCloudManagerPool-1723-thread-3) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
12
   [junit4]   2> 1060489 DEBUG (simCloudManagerPool-1723-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060489 DEBUG (simCloudManagerPool-1723-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060489 DEBUG (simCloudManagerPool-1723-thread-3) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated 
with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction},
 {name=test1, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}],
 enabled=true}
   [junit4]   2> 1060499 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
12
   [junit4]   2> 1060499 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
   [junit4]   2> 1060499 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1060499 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1060499 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 
973906891914713
   [junit4]   2> 1060499 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1060536 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1060536 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1060536 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10004_solr] added at times 
[973906891914713], now=973908706136163
   [junit4]   2> 1060536 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375c35e563dd9Te19v0m1w0ylx6923iq76qlam3",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973906891914713,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973906891914713],
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1060552 DEBUG (ScheduledTrigger-1724-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 1060613 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375c35e563dd9Te19v0m1w0ylx6923iq76qlam3",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973906891914713,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973906891914713],
   [junit4]   2>     "_enqueue_time_":973909515570463,
   [junit4]   2>     "nodeNames":["127.0.0.1:10004_solr"]}}
   [junit4]   2> 1060613 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 1060613 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 1060614 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 46 ms for event 
id=375c35e563dd9Te19v0m1w0ylx6923iq76qlam3
   [junit4]   2> 1060718 DEBUG (ScheduledTrigger-1724-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1060718 DEBUG (ScheduledTrigger-1724-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1060718 DEBUG (ScheduledTrigger-1724-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1060718 DEBUG (ScheduledTrigger-1724-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10005_solr at time 
973917843364913
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 4
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing 
registered processor for nodes: [127.0.0.1:10005_solr] added at times 
[973917843364913], now=973918862876363
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375c5eb181c31Te19v0m1w0ylx6923iq76qlam6",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973917843364913,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973917843364913],
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 1060739 DEBUG (ScheduledTrigger-1724-thread-2) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_trigger]
   [junit4]   2> 1060771 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375c5eb181c31Te19v0m1w0ylx6923iq76qlam6",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973917843364913,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973917843364913],
   [junit4]   2>     "_enqueue_time_":973918872844863,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2> 1060791 WARN  (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Exception executing actions
   [junit4]   2> java.lang.Exception: Error executing action: test1 for trigger 
event: {
   [junit4]   2>   "id":"375c5eb181c31Te19v0m1w0ylx6923iq76qlam6",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":973917843364913,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973917843364913],
   [junit4]   2>     "_enqueue_time_":973918872844863,
   [junit4]   2>     "nodeNames":["127.0.0.1:10005_solr"]}}
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:302)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.RuntimeException: failure
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:890)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:299)
   [junit4]   2>        ... 6 more
   [junit4]   2> 1060791 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 1060791 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
   [junit4]   2> 1060791 DEBUG (AutoscalingActionExecutor-1725-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 1028 ms for event 
id=375c5eb181c31Te19v0m1w0ylx6923iq76qlam6
   [junit4]   2> 1060815 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testListeners
   [junit4]   2> 1060815 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 1060815 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase #############################################
   [junit4]   2> 
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:               4
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:      4
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:               0
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:      []
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:    0
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:    0
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:           0
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - autoscaling        3
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - update             4
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - FAILED          1
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - STARTED         2
   [junit4]   2> 1060848 INFO  
(TEST-TestTriggerIntegration.testListeners-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimSolrCloudTestCase ##           - SUCCEEDED       1
   [junit4]   2> 1060851 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
   [junit4]   2> 1060851 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 13
   [junit4]   2> 1060864 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
13
   [junit4]   2> 1060864 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 1060865 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1060865 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
   [junit4]   2> 1060884 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and 
clearing object cache...
   [junit4]   2> 1060884 DEBUG 
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed 
explicitly
   [junit4]   2> 1060900 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1060921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1060921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 14
   [junit4]   2> 1060921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1060921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion -1
   [junit4]   2> 1060921 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
14
   [junit4]   2> 1060937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1060937 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
   [junit4]   2> 1060937 DEBUG (ScheduledTrigger-1727-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1060957 DEBUG (ScheduledTrigger-1727-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061010 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061030 DEBUG (ScheduledTrigger-1727-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061031 DEBUG (simCloudManagerPool-1726-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1061031 DEBUG (simCloudManagerPool-1726-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
15
   [junit4]   2> 1061031 DEBUG (simCloudManagerPool-1726-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061031 DEBUG (simCloudManagerPool-1726-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1061031 DEBUG (simCloudManagerPool-1726-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061031 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
15
   [junit4]   2> 1061031 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1061031 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
   [junit4]   2> 1061031 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1061031 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 1061066 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1061066 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1061066 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1061066 DEBUG (ScheduledTrigger-1727-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10006_solr at time 
973935249390013
   [junit4]   2> 1061067 DEBUG (simCloudManagerPool-1726-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1061067 DEBUG (simCloudManagerPool-1726-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
16
   [junit4]   2> 1061067 DEBUG (simCloudManagerPool-1726-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 1061067 DEBUG (simCloudManagerPool-1726-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger 
instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1061067 DEBUG (simCloudManagerPool-1726-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
   [junit4]   2> 1061067 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
16
   [junit4]   2> 1061068 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1061068 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10006_solr
   [junit4]   2> 1061068 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
   [junit4]   2> 1061068 DEBUG (ScheduledTrigger-1727-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
   [junit4]   2> 1061068 DEBUG (ScheduledTrigger-1727-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 1061068 DEBUG (ScheduledTrigger-1727-thread-4) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing 
registered processor for nodes: [127.0.0.1:10006_solr] added at times 
[973935318853863], now=973935326199513
   [junit4]   2> 1061068 DEBUG (ScheduledTrigger-1727-thread-4) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375c9fcb6e8e7Te19v0m1w0ylx6923iq76qlam9",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":973935318853863,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973935318853863],
   [junit4]   2>     "nodeNames":["127.0.0.1:10006_solr"]}}
   [junit4]   2> 1061068 DEBUG (ScheduledTrigger-1727-thread-4) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_added_restore_trigger]
   [junit4]   2> 1061069 DEBUG (AutoscalingActionExecutor-1728-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375c9fcb6e8e7Te19v0m1w0ylx6923iq76qlam9",
   [junit4]   2>   "source":"node_added_restore_trigger",
   [junit4]   2>   "eventTime":973935318853863,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[973935318853863],
   [junit4]   2>     "_enqueue_time_":973935337965513,
   [junit4]   2>     "nodeNames":["127.0.0.1:10006_solr"]}}
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS 
############
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:            3
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:   3
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:            0
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:   []
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:                0
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts 
##########
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##                - autoscaling        
2
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##                - update             
1
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: 
node_added_restore_trigger
   [junit4]   2> 1061069 INFO  
(TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##                - STARTED         1
   [junit4]   2> 1061197 INFO  
(TEST-TestTriggerIntegration.testCooldown-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testCooldown
   [junit4]   2> 1061197 DEBUG 
(TEST-TestTriggerIntegration.testCooldown-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
17
   [junit4]   2> 1061197 INFO  
(TEST-TestTriggerIntegration.testCooldown-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 1061197 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1061212 WARN  (AutoscalingActionExecutor-1728-thread-1) [    ] 
o.a.s.c.a.SystemLogListener Exception sending event to collection .system
   [junit4]   2> java.io.IOException: java.lang.InterruptedException
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:460)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.SystemLogListener.onEvent(SystemLogListener.java:122)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:744)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerListeners.fireListeners(ScheduledTriggers.java:714)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:312)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.InterruptedException
   [junit4]   2>        at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.get(FutureTask.java:191)
   [junit4]   2>        at 
org.apache.solr.cloud.autoscaling.sim.SimCloudManager.request(SimCloudManager.java:458)
   [junit4]   2>        ... 10 more
   [junit4]   2> 1061213 DEBUG (AutoscalingActionExecutor-1728-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 7203 ms for event 
id=375c9fcb6e8e7Te19v0m1w0ylx6923iq76qlam9
   [junit4]   2> 1061213 DEBUG 
(TEST-TestTriggerIntegration.testCooldown-seed#[92DAA5CA5D099EAA]) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 1061253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1061253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 18
   [junit4]   2> 1061253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion -1
   [junit4]   2> 1061253 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
18
   [junit4]   2> 1061257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1061257 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
   [junit4]   2> 1061257 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061293 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061342 DEBUG (ScheduledTrigger-1730-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061366 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061375 DEBUG (simCloudManagerPool-1729-thread-1) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1061375 DEBUG (simCloudManagerPool-1729-thread-1) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
19
   [junit4]   2> 1061415 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1061431 DEBUG (simCloudManagerPool-1729-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061431 DEBUG (simCloudManagerPool-1729-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger 
instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1061431 DEBUG (simCloudManagerPool-1729-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061435 DEBUG (simCloudManagerPool-1729-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1061439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
19
   [junit4]   2> 1061439 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1061439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
   [junit4]   2> 1061447 DEBUG (simCloudManagerPool-1729-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
20
   [junit4]   2> 1061447 DEBUG (simCloudManagerPool-1729-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061447 DEBUG (simCloudManagerPool-1729-thread-2) [    ] 
o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger 
instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, 
class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}],
 enabled=true}
   [junit4]   2> 1061447 DEBUG (simCloudManagerPool-1729-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 
127.0.0.1:10003_solr]
   [junit4]   2> 1061451 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 3
   [junit4]   2> 1061451 DEBUG (ScheduledTrigger-1730-thread-1) [    ] 
o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrig

[...truncated too long message...]

2> 1065610 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] 
o.a.s.c.a.OverseerTriggerThread Current znodeVersion 43, lastZnodeVersion 43
   [junit4]   2> 1065610 DEBUG (ScheduledTrigger-1754-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 1065610 DEBUG (ScheduledTrigger-1754-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10015_solr
   [junit4]   2> 1065611 DEBUG (ScheduledTrigger-1754-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1065631 DEBUG (ScheduledTrigger-1754-thread-1) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 1065631 DEBUG (ScheduledTrigger-1754-thread-4) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with 
currently live nodes: 2
   [junit4]   2> 1065641 DEBUG (simCloudManagerPool-1753-thread-2) [    ] 
o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 1065641 DEBUG (simCloudManagerPool-1753-thread-2) [    ] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
44
   [junit4]   2> 1065641 DEBUG (simCloudManagerPool-1753-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr, 
127.0.0.1:10013_solr]
   [junit4]   2> 1065641 DEBUG (simCloudManagerPool-1753-thread-2) [    ] 
o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr, 
127.0.0.1:10013_solr]
   [junit4]   2> 1065641 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 
44
   [junit4]   2> 1065641 INFO  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
   [junit4]   2> 1065641 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1065641 DEBUG (ScheduledTrigger-1754-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger 
with currently live nodes: 2
   [junit4]   2> 1065641 DEBUG (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 44, lastZnodeVersion 44
   [junit4]   2> 1065641 DEBUG (ScheduledTrigger-1754-thread-3) [    ] 
o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost 
nodes: [127.0.0.1:10015_solr]
   [junit4]   2> 1065641 DEBUG (ScheduledTrigger-1754-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"375fedda9af95Te19v0m1w0ylx6923iq76qlana",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":974162431160213,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[974162431160213],
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1065641 DEBUG (ScheduledTrigger-1754-thread-3) [    ] 
o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, 
node_lost_restore_trigger]
   [junit4]   2> 1065642 DEBUG (AutoscalingActionExecutor-1755-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"375fedda9af95Te19v0m1w0ylx6923iq76qlana",
   [junit4]   2>   "source":"node_lost_restore_trigger",
   [junit4]   2>   "eventTime":974162431160213,
   [junit4]   2>   "eventType":"NODELOST",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[974162431160213],
   [junit4]   2>     "_enqueue_time_":974163997670163,
   [junit4]   2>     "nodeNames":["127.0.0.1:10015_solr"]}}
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
   [junit4]   2> 
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS 
############
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase 
#############################################
   [junit4]   2> 
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes:             2
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes:    2
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes:             1
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         127.0.0.1:10015_solr
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections:    []
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node:  0
   [junit4]   2> 1065642 DEBUG (AutoscalingActionExecutor-1755-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node:  0
   [junit4]   2> 1065642 DEBUG (AutoscalingActionExecutor-1755-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after 
100ms
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas:         0
   [junit4]   2> 1065642 DEBUG (AutoscalingActionExecutor-1755-thread-1) [    ] 
o.a.s.c.a.ScheduledTriggers -- processing took 23 ms for event 
id=375fedda9af95Te19v0m1w0ylx6923iq76qlana
   [junit4]   2> 1065642 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts 
##########
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         - autoscaling        2
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         - update             2
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts 
###########
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_restore_trigger
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         - STARTED         1
   [junit4]   2> 1065643 INFO  
(TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[92DAA5CA5D099EAA])
 [    ] o.a.s.c.a.s.SimSolrCloudTestCase ##         - SUCCEEDED       1
   [junit4]   2> 1065643 WARN  (Simulated OverseerAutoScalingTriggerThread) [   
 ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are 
closed, exiting.
   [junit4]   2> 1065643 DEBUG 
(SUITE-TestTriggerIntegration-seed#[92DAA5CA5D099EAA]-worker) [    ] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_92DAA5CA5D099EAA-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=1770, maxMBSortInHeap=7.503565718867579, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@44cfe137),
 locale=bg, timezone=Asia/Phnom_Penh
   [junit4]   2> NOTE: Linux 4.4.0-104-generic amd64/Oracle Corporation 
1.8.0_152 (64-bit)/cpus=4,threads=1,free=158582152,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, 
TestManagedSchemaAPI, SpellCheckComponentTest, EnumFieldTest, 
MultiThreadedOCPTest, ClusterStateTest, TestSha256AuthenticationProvider, 
AddBlockUpdateTest, TestSolrCloudWithHadoopAuthPlugin, HdfsLockFactoryTest, 
AnalysisAfterCoreReloadTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, 
IndexSchemaRuntimeFieldTest, TestDistributedMissingSort, 
MoveReplicaHDFSFailoverTest, SolrShardReporterTest, 
TestDistributedStatsComponentCardinality, RecoveryZkTest, TestLFUCache, 
TestStressReorder, CollectionsAPISolrJTest, TestMacroExpander, 
DistanceFunctionTest, TestQueryTypes, TestCoreDiscovery, 
TestDocBasedVersionConstraints, UnloadDistributedZkTest, 
NotRequiredUniqueKeyTest, DirectoryFactoryTest, TestAnalyzeInfixSuggestions, 
TestNumericTerms32, TestTriggerIntegration]
   [junit4] Completed [118/768 (1!)] on J2 in 8.58s, 11 tests, 2 failures <<< 
FAILURES!

[...truncated 50261 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to