No IO issues and it runs on SSD. Machine is also stable and has no SATA timeouts or similar stuff.
It is just a 3 year old server CPU and its running a Vbox VM in parallel. Uwe On 15. März 2014 21:31:10 MEZ, Mark Miller <[email protected]> wrote: >Hmm…only interesting logging I see is this: > >57473 T32 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in >SyncThread:0 took 50531ms which will adversely effect operation >latency. See the ZooKeeper troubleshooting guide >I wonder if that means that if i boost the connect timeout from 45 to >60 seconds, it will pass. >Perhaps this machine has some IO issues? > >-- >Mark Miller >about.me/markrmiller > >On March 15, 2014 at 9:23:25 AM, Policeman Jenkins Server >([email protected]) wrote: > >Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9800/ >Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC > >1 tests failed. >REGRESSION: >org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch > > >Error Message: >java.util.concurrent.TimeoutException: Could not connect to ZooKeeper >127.0.0.1:44565 within 45000 ms > >Stack Trace: >org.apache.solr.common.SolrException: >java.util.concurrent.TimeoutException: Could not connect to ZooKeeper >127.0.0.1:44565 within 45000 ms >at >__randomizedtesting.SeedInfo.seed([D09CC97019C4AF45:517A47686E9BCF79]:0) > >at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150) > >at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101) > >at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91) > >at >org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89) > >at >org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83) > >at >org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70) > >at >org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201) > >at >org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78) > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >at java.lang.reflect.Method.invoke(Method.java:606) >at >com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876) > >at >com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) > >at >org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) > >at >org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) > >at >org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > >at >com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > >at >org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) > >at >org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) > >at >org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > >at >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >at >com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359) > >at >com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783) > >at >com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771) > >at >com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782) > >at >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >at >com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) > >at >org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > >at >org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) > >at >com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > >at >com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > >at >com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > >at >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >at >org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) > >at >org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > >at >org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) > >at >org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) > >at >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >at >com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359) > >at java.lang.Thread.run(Thread.java:744) >Caused by: java.util.concurrent.TimeoutException: Could not connect to >ZooKeeper 127.0.0.1:44565 within 45000 ms >at >org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223) > >at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142) > >... 45 more > > > > >Build Log: >[...truncated 11677 lines...] >[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest >[junit4] 2> 6735 T28 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl >(false) and clientAuth (false) >[junit4] 2> 6736 T28 oas.BaseDistributedSearchTestCase.initHostContext >Setting hostContext system property: / >[junit4] 2> 6737 T28 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() >does not exist - solrj test? >[junit4] 2> 6747 T28 oas.SolrTestCaseJ4.setUp ###Starting >testDistribSearch >[junit4] 2> Creating dataDir: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1394889608728 > >[junit4] 2> 6750 T28 oasc.ZkTestServer.run STARTING ZK TEST SERVER >[junit4] 2> 6752 T29 oasc.ZkTestServer$ZKServerMain.runFromConfig >Starting server >[junit4] 2> 6852 T28 oasc.ZkTestServer.run start zk server on >port:44565 >[junit4] 2> 6896 T28 oascc.ConnectionManager.waitForConnected Waiting >for client to connect to ZooKeeper >[junit4] 2> 57473 T32 oazsp.FileTxnLog.commit WARN fsync-ing the write >ahead log in SyncThread:0 took 50531ms which will adversely effect >operation latency. See the ZooKeeper troubleshooting guide >[junit4] 2> 57474 T28 oas.SolrTestCaseJ4.tearDown ###Ending >testDistribSearch >[junit4] 2> 57491 T30 oazs.NIOServerCnxn.doIO WARN caught end of stream >exception EndOfStreamException: Unable to read additional data from >client sessionid 0x144c5e64e5b0000, likely client has closed socket >[junit4] 2> at >org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) > >[junit4] 2> at >org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > >[junit4] 2> at java.lang.Thread.run(Thread.java:744) >[junit4] 2> >[junit4] 2> 57492 T28 oasc.ZkTestServer.send4LetterWord connecting to >127.0.0.1:44565 44565 >[junit4] 1> client port:0.0.0.0/0.0.0.0:0 >[junit4] 2> NOTE: reproduce with: ant test >-Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch >-Dtests.seed=D09CC97019C4AF45 -Dtests.multiplier=3 -Dtests.slow=true >-Dtests.locale=sl -Dtests.timezone=America/Argentina/Rio_Gallegos >-Dtests.file.encoding=US-ASCII >[junit4] ERROR 51.1s J0 | CloudSolrServerTest.testDistribSearch <<< >[junit4] > Throwable #1: org.apache.solr.common.SolrException: >java.util.concurrent.TimeoutException: Could not connect to ZooKeeper >127.0.0.1:44565 within 45000 ms >[junit4] > at >__randomizedtesting.SeedInfo.seed([D09CC97019C4AF45:517A47686E9BCF79]:0) > >[junit4] > at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150) > >[junit4] > at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101) > >[junit4] > at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91) > >[junit4] > at >org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89) > >[junit4] > at >org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83) > >[junit4] > at >org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70) > >[junit4] > at >org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201) > >[junit4] > at >org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78) > >[junit4] > at java.lang.Thread.run(Thread.java:744) >[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not >connect to ZooKeeper 127.0.0.1:44565 within 45000 ms >[junit4] > at >org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223) > >[junit4] > at >org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142) > >[junit4] > ... 45 more >[junit4] 2> 57872 T28 oas.SolrTestCaseJ4.setUp ###Starting testShutdown > >[junit4] 2> Creating dataDir: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1394889659853 > >[junit4] 2> 57873 T28 oasc.ZkTestServer.run STARTING ZK TEST SERVER >[junit4] 2> 57873 T36 oasc.ZkTestServer$ZKServerMain.runFromConfig >Starting server >[junit4] 2> 57973 T28 oasc.ZkTestServer.run start zk server on >port:36992 >[junit4] 2> 57974 T28 oascc.ConnectionManager.waitForConnected Waiting >for client to connect to ZooKeeper >[junit4] 2> 57984 T42 oascc.ConnectionManager.process Watcher >org.apache.solr.common.cloud.ConnectionManager@f83b4c >name:ZooKeeperConnection Watcher:127.0.0.1:36992 got event WatchedEvent >state:SyncConnected type:None path:null path:null type:None >[junit4] 2> 57985 T28 oascc.ConnectionManager.waitForConnected Client >is connected to ZooKeeper >[junit4] 2> 57986 T28 oascc.SolrZkClient.makePath makePath: /solr >[junit4] 2> 58008 T28 oascc.ConnectionManager.waitForConnected Waiting >for client to connect to ZooKeeper >[junit4] 2> 58009 T44 oascc.ConnectionManager.process Watcher >org.apache.solr.common.cloud.ConnectionManager@62c054 >name:ZooKeeperConnection Watcher:127.0.0.1:36992/solr got event >WatchedEvent state:SyncConnected type:None path:null path:null >type:None >[junit4] 2> 58010 T28 oascc.ConnectionManager.waitForConnected Client >is connected to ZooKeeper >[junit4] 2> 58013 T28 oascc.SolrZkClient.makePath makePath: >/collections/collection1 >[junit4] 2> 58017 T28 oascc.SolrZkClient.makePath makePath: >/collections/collection1/shards >[junit4] 2> 58021 T28 oascc.SolrZkClient.makePath makePath: >/collections/control_collection >[junit4] 2> 58024 T28 oascc.SolrZkClient.makePath makePath: >/collections/control_collection/shards >[junit4] 2> 58028 T28 oasc.AbstractZkTestCase.putConfig put >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml >to /configs/conf1/solrconfig.xml >[junit4] 2> 58029 T28 oascc.SolrZkClient.makePath makePath: >/configs/conf1/solrconfig.xml >[junit4] 2> 58037 T28 oasc.AbstractZkTestCase.putConfig put >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml >to /configs/conf1/schema.xml >[junit4] 2> 58037 T28 oascc.SolrZkClient.makePath makePath: >/configs/conf1/schema.xml >[junit4] 2> 58155 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml >because it doesn't exist >[junit4] 2> 58156 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt >because it doesn't exist >[junit4] 2> 58157 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt >because it doesn't exist >[junit4] 2> 58158 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml >because it doesn't exist >[junit4] 2> 58159 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml >because it doesn't exist >[junit4] 2> 58161 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json >because it doesn't exist >[junit4] 2> 58161 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt >because it doesn't exist >[junit4] 2> 58162 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt >because it doesn't exist >[junit4] 2> 58163 T28 oasc.AbstractZkTestCase.putConfig skipping >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt >because it doesn't exist >[junit4] 2> 58182 T28 oascc.ConnectionManager.waitForConnected Waiting >for client to connect to ZooKeeper >[junit4] 2> 58184 T45 oaz.ClientCnxnSocketNIO.connect ERROR Unable to >open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332 >[junit4] 2> 58184 T45 oaz.ClientCnxn$SendThread.run WARN Session 0x0 >for server null, unexpected error, closing socket connection and >attempting reconnect java.net.SocketException: Network is unreachable >[junit4] 2> at sun.nio.ch.Net.connect0(Native Method) >[junit4] 2> at sun.nio.ch.Net.connect(Net.java:465) >[junit4] 2> at sun.nio.ch.Net.connect(Net.java:457) >[junit4] 2> at >sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) >[junit4] 2> at >org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266) > >[junit4] 2> at >org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276) > >[junit4] 2> at >org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958) > >[junit4] 2> at >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) >[junit4] 2> >[junit4] 2> 59286 T45 oaz.ClientCnxnSocketNIO.connect ERROR Unable to >open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332 >[junit4] 2> 59389 T28 oas.SolrTestCaseJ4.tearDown ###Ending >testShutdown >[junit4] 2> 59390 T28 oasc.ZkTestServer.send4LetterWord connecting to >127.0.0.1:36992 36992 >[junit4] 1> client port:0.0.0.0/0.0.0.0:0 >[junit4] 2> 64532 T28 oas.SolrTestCaseJ4.deleteCore ###deleteCore >[junit4] 2> NOTE: test params are: codec=Lucene42, >sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, >locale=sl, timezone=America/Argentina/Rio_Gallegos >[junit4] 2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation >1.7.0_51 (32-bit)/cpus=8,threads=1,free=10223216,total=33554432 >[junit4] 2> NOTE: All tests run in this JVM: [TestXMLEscaping, >TermsResponseTest, TestJavaBinCodec, SolrExceptionTest, SolrPingTest, >TestUpdateRequestCodec, CloudSolrServerTest] >[junit4] Completed on J0 in 57.84s, 2 tests, 1 error <<< FAILURES! > >[...truncated 121 lines...] >BUILD FAILED >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The >following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The >following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The >following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: >The following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: >The following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: >The following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: >The following error occurred while executing this line: >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: >There were test failures: 49 suites, 283 tests, 1 error > >Total time: 53 minutes 33 seconds >Build step 'Invoke Ant' marked build as failure >Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC >Archiving artifacts >Recording test results >Email was triggered for: Failure >Sending email for trigger: Failure > > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: [email protected] >For additional commands, e-mail: [email protected] -- Uwe Schindler H.-H.-Meier-Allee 63, 28213 Bremen http://www.thetaphi.de
