[
https://issues.apache.org/jira/browse/SOLR-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234906#comment-14234906
]
Hoss Man commented on SOLR-6675:
--------------------------------
Forest: if there is an Object "wait" happening in getSearcher, that means some
other thread is already warming up a searcher - a full thread dump of every
thread would be helpful to see why/how.
it looks like you are seeing the perfect storm of 3 problems:
* getStatistics is being called when *registering* the MBean
* getStatistics is blocking because it needs the searcher and the searcher is
being warmed
* something is making your searcher take 10 minutes to warm
> Solr webapp deployment is very slow with <jmx/> in solrconfig.xml
> -----------------------------------------------------------------
>
> Key: SOLR-6675
> URL: https://issues.apache.org/jira/browse/SOLR-6675
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.7
> Environment: Linux Redhat 64bit
> Reporter: Forest Soup
> Priority: Critical
> Labels: performance
> Attachments: callstack.png
>
>
> We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr
> index(cores) are big(50~100G) each core.
> When we start up tomcat, the solr webapp deployment is very slow. From
> tomcat's catalina log, every time it takes about 10 minutes to get deployed.
> After we analyzing java core dump, we notice it's because the loading process
> cannot finish until the MBean calculation for large index is done.
>
> So we tried to remove the <jmx/> from solrconfig.xml, after that, the loading
> of solr webapp only take about 1 minute. So we can sure the MBean calculation
> for large index is the root cause.
> Could you please point me if there is any async way to do statistic
> monitoring without <jmx/> in solrconfig.xml, or let it do calculation after
> the deployment? Thanks!
> The callstack.png file in the attachment is the call stack of the long
> blocking thread which is doing statistics calculation.
> The catalina log of tomcat:
> INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
> Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR
> INFO: Deploying web application archive
> /opt/ibm/solrsearch/tomcat/webapps/solr.war
> Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR
> INFO: Deployment of web application archive
> /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms
> <---------------- Time taken for solr app Deployment is about 10 minutes
> -----------
> Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory
> /opt/ibm/solrsearch/tomcat/webapps/manager
> Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deployment of web application directory
> /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms
> Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory
> /opt/ibm/solrsearch/tomcat/webapps/examples
> Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deployment of web application directory
> /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms
> Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory
> /opt/ibm/solrsearch/tomcat/webapps/docs
> Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deployment of web application directory
> /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms
> Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory
> /opt/ibm/solrsearch/tomcat/webapps/ROOT
> Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deployment of web application directory
> /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms
> Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deploying web application directory
> /opt/ibm/solrsearch/tomcat/webapps/host-manager
> Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory
> INFO: Deployment of web application directory
> /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms
> Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["http-bio-8080"]
> Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["ajp-bio-8009"]
> Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 601506 ms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
