Hi all,
while I was testing out the CertAuthPlugin for the new SolR 9 it came to my attention that various internal HTTP calls in SolR fail. For example when I try to add a BinaryResponseWriter via curl it fails with lots of authentication errors (HTTP status code 401). Other actions (like creating schema fields for collections) via curl work fine. To reproduce the problem, following steps have to be taken (on Linux): - git clone https://github.com/apache/solr.git (I used commit caf8cbc0aa11e32f894a90531e3e9f20edf75efa) - cd solr - ./gradlew assemble - cd solr/packaging/build/solr-9.0.0-SNAPSHOT/ - keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:127.0.0.1 -dname "CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country" - openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.key -nodes -nocerts - openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.crt -nodes -nokeys - echo 'SOLR_SSL_ENABLED=true' >> bin/solr.in.sh - echo 'SOLR_SSL_KEY_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh - echo 'SOLR_SSL_KEY_STORE_PASSWORD=secret' >> bin/solr.in.sh - echo 'SOLR_SSL_TRUST_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh - echo 'SOLR_SSL_TRUST_STORE_PASSWORD=secret' >> bin/solr.in.sh - echo 'SOLR_SSL_NEED_CLIENT_AUTH=true' >> bin/solr.in.sh - echo 'SOLR_SSL_WANT_CLIENT_AUTH=false' >> bin/solr.in.sh - echo 'SOLR_SSL_CHECK_PEER_NAME=false' >> bin/solr.in.sh - echo '{ "authentication": { "class": "org.apache.solr.security.CertAuthPlugin" }, "authorization": { "class": "solr.RuleBasedAuthorizationPlugin", "permissions": [ { "name": "all", "role": [ "admin-role" ] } ], "user-role": { "CN=localhost,OU=Organizational Unit,O=Organization,L=Location,ST=State,C=Country": [ "admin-role"] } } }' > /tmp/security.json - ./bin/solr start -v -c - server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd clusterprop -name urlScheme -val https - ./bin/solr zk cp file:///tmp/security.json zk:/security.json -z localhost:9983 - ./bin/solr stop - ./bin/solr start -v -c - ./bin/solr create -c testcollection - curl --cacert ./solr-ssl.keystore.crt --key ./solr-ssl.keystore.key --cert ./solr-ssl.keystore.crt "https://localhost:8983/api/collections/testcollection/config"; -H "Content-Type: application/json" --data-binary '{ "add-queryresponsewriter":{ "class":"solr.BinaryResponseWriter", "name":"test" }}' After the last curl command (which takes about 30 seconds) the following error message is printed: { "responseHeader":{ "status":500, "QTime":30017}, "errorMessages":["1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://localhost:8983/solr/testcollection_shard1_replica_n1/]\n";], "WARNING":"This response format is experimental. It is likely to change in the future.", "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://localhost:8983/solr/testcollection_shard1_replica_n1/]";, "trace":"org.apache.solr.common.SolrException: 1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://localhost:8983/solr/testcollection_shard1_replica_n1/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:829)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:549)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:381)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:140 )\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)\n\tat org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:269)\n\tat org.apache.solr.api.V2HttpCall.execute(V2HttpCall.java:354)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat org.eclipse.jetty.servlet. ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)\n\tat org.eclipse.j etty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:134 9)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.hand ler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.jav a:105)\n\tat org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:146)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n", "code":500}} In the SolR WEB-UI the following errors are printed: If required I will gladly send the full debug log of the server; it´s compressed about 500 kb in size. The system where this happens is a CentOS 7 with JDK 11 installed. Out of curiosity I backported the CertAuthPlugin to SolR v8.8.1 locally and SolR 8 had the same issues as the current master. I wonder if this is a bug or if I did some misconfiguration here. Thanks & Greetings, Dominik
