[
https://issues.apache.org/jira/browse/SOLR-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Darren Spehr closed SOLR-6995.
------------------------------
Resolution: Duplicate
> On a new cluster setup, new nodes never move out of the 'down' state.
> ---------------------------------------------------------------------
>
> Key: SOLR-6995
> URL: https://issues.apache.org/jira/browse/SOLR-6995
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.3
> Reporter: Darren Spehr
>
> This is related to a question I posted on
> [stackoverflow|http://stackoverflow.com/questions/28004832/solr-4-10-3-is-not-proceeding-to-leader-election-on-new-cluster-startup-hangs]
> a day ago.
> When deploying a new cluster, new nodes never proceed to an 'active' state. I
> have used both a custom (yet simple) deployment, as well as using the example
> framework you get when downloading the Solr distribution. Further inspection
> of the logs, and comparing them to the output of a 4.10.1 start up, it looks
> like the nodes never move on to leader election.
> For an apples-to-apples comparison I moved the solr/collection1 directory to
> solr/ttPoiMDS, and modified the core.properties appropriately. I also removed
> the 'conf' directory, assuming that it would pick this up from our ZooKeeper
> cluster.
> Here is the start up:
> {code}
> java
> -Dcollection.configName=ttPoiMDS
> -Djava.util.logging.config.file=etc/logging.properties
> -DzkHost=our_zk_host_1:<zk_port>
> -Djetty.port=8983
> -jar start.jar
> {code}
> Here is the output in the logs:
> {code}
> 0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
> 32 [main] INFO org.eclipse.jetty.deploy.providers.ScanningAppProvider –
> Deployment monitor /site/pkgs/solr/solr-4.10.3/example/contexts at interval 0
> 38 [main] INFO org.eclipse.jetty.deploy.DeploymentManager – Deployable
> added: /site/pkgs/solr/solr-4.10.3/example/contexts/solr-jetty-context.xml
> 114 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration – Extract
> jar:file:/site/pkgs/solr/solr-4.10.3/example/webapps/solr.war!/ to
> /site/pkgs/solr/solr-4.10.3/example/solr-webapp/webapp
> 1527 [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor – NO
> JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
> 1597 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> SolrDispatchFilter.init()
> 1616 [main] INFO org.apache.solr.core.SolrResourceLoader – JNDI not
> configured for solr (NoInitialContextEx)
> 1616 [main] INFO org.apache.solr.core.SolrResourceLoader – solr home
> defaulted to 'solr/' (could not find system property or JNDI)
> 1617 [main] INFO org.apache.solr.core.SolrResourceLoader – new
> SolrResourceLoader for directory: 'solr/'
> 1748 [main] INFO org.apache.solr.core.ConfigSolr – Loading container
> configuration from /site/pkgs/solr/solr-4.10.3/example/solr/solr.xml
> 1884 [main] INFO org.apache.solr.core.CoresLocator – Config-defined core
> root directory: /site/pkgs/solr/solr-4.10.3/example/solr
> 1890 [main] INFO org.apache.solr.core.CoreContainer – New CoreContainer
> 378496804
> 1890 [main] INFO org.apache.solr.core.CoreContainer – Loading cores into
> CoreContainer [instanceDir=solr/]
> 1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting socketTimeout to: 0
> 1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting urlScheme to: null
> 1909 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting connTimeout to: 0
> 1911 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maxConnectionsPerHost to: 20
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting corePoolSize to: 0
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maximumPoolSize to: 2147483647
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maxThreadIdleTime to: 5
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting sizeOfQueue to: -1
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting fairnessPolicy to: false
> 2054 [main] INFO org.apache.solr.update.UpdateShardHandler – Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=0&connTimeout=0&retry=false
> 2056 [main] INFO org.apache.solr.logging.LogWatcher – SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> 2057 [main] INFO org.apache.solr.logging.LogWatcher – Registering Log
> Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> 2058 [main] INFO org.apache.solr.core.CoreContainer – Host Name:
> 2059 [main] INFO org.apache.solr.core.ZkContainer – Zookeeper
> client=our_zk_host_1:2195
> 2140 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Waiting
> for client to connect to ZooKeeper
> 2220 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager
> – Watcher org.apache.solr.common.cloud.ConnectionManager@36dadde6
> name:ZooKeeperConnection Watcher:our_zk_host_1:2195 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> 2220 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Client is
> connected to ZooKeeper
> 2293 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/queue
> 2447 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-queue-work
> 2609 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-running
> 2758 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-completed
> 2919 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-failure
> 3093 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /live_nodes
> 3173 [main] INFO org.apache.solr.cloud.ZkController – Register node as live
> in ZooKeeper:/live_nodes/our_solr_host:8983_solr
> 3201 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /live_nodes/our_solr_host:8983_solr
> 3371 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect
> 3477 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect/election
> 3577 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=null) closing
> 3659 [main] INFO org.apache.solr.cloud.ElectionContext – I am going to be
> the leader our_solr_host:8983_solr
> 3662 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect/leader
> 3763 [main] INFO org.apache.solr.cloud.Overseer – Overseer
> (id=93155855038349319-our_solr_host:8983_solr-n_0000000000) starting
> 3887 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/queue-work
> 4409 [main] INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread –
> Starting OverseerAutoReplicaFailoverThread
> autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
> 4438
> [OverseerCollectionProcessor-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor – Process current
> queue of collection creations
> 4456 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /clusterstate.json
> 4559 [main] INFO org.apache.solr.common.cloud.ZkStateReader – Updating
> cluster state from ZooKeeper...
> 4731
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Starting to work on the main queue
> 4736 [main] INFO org.apache.solr.core.CoresLocator – Looking for core
> definitions underneath /site/pkgs/solr/solr-4.10.3/example/solr
> 4744 [main] INFO org.apache.solr.core.CoresLocator – Found core ttPoiMDS in
> /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/
> 4744 [main] INFO org.apache.solr.core.CoresLocator – Found 1 core
> definitions
> 4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – publishing core=ttPoiMDS state=down collection=ttPoiMDS
> 4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – numShards not found on descriptor - reading it from system property
> 4796 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – look for our core node name
> 4797 [zkCallback-2-thread-1] INFO org.apache.solr.cloud.DistributedQueue –
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 4897
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
> "operation":"state",
> "shard":null,
> "roles":null,
> "state":"down",
> "core":"ttPoiMDS",
> "collection":"ttPoiMDS",
> "node_name":"our_solr_host:8983_solr",
> "base_url":"http://our_solr_host:8983/solr"}
> 4899
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Assigning new node to shard
> shard=shard1
> 5052 [zkCallback-2-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader
> – A cluster state change: WatchedEvent state:SyncConnected
> type:NodeDataChanged path:/clusterstate.json, has occurred - updating...
> (live nodes size: 1)
> 5800 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – waiting to find shard id in clusterstate for ttPoiMDS
> 5801 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – Check for collection zkNode:ttPoiMDS
> 5924 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – Collection zkNode exists
> 5925 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.common.cloud.ZkStateReader – Load collection config
> from:/collections/ttPoiMDS
> 5970 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.common.cloud.ZkStateReader – path=/collections/ttPoiMDS
> configName=ttPoiMDS specified config exists in ZooKeeper
> 5970 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for
> directory: '/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/'
> 6054 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
> Adding specified lib dirs to ClassLoader
> 6055 [coreLoadExecutor-6-thread-1] WARN
> org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to
> add to classloader: ../../lib/mq (resolved as:
> /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/../../lib/mq).
> 6139 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
> Using Lucene MatchVersion: 4.10.3
> 6238 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.Config –
> Loaded SolrConfig: solrconfig.xml
> 6320 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema
> – Reading Solr Schema from /configs/ttPoiMDS/schema.xml
> 6347 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema
> – [ttPoiMDS] Schema name=poiMDS
> 6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> user.dir=/site/pkgs/solr/solr-4.10.3/example
> 6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> SolrDispatchFilter.init() done
> 6614 [main] INFO org.eclipse.jetty.server.AbstractConnector – Started
> [email protected]:8983
> {code}
> Startup then seems to hang. The Solr UI is available, but it shows that no
> cores are available. In the cloud view the configuration from ZooKeeper is
> present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
