Really could do with some insight on this. One of the nodes on the server reports constant connection errors in and out.
I see things like: 2025-02-05 10:03:47.518 WARN (qtp1959239586-70) [] o.a.s.h.a.AdminHandlersProxy Exception when fetching result from node GIG01SRVWCRSOL1:18983_solr => java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://GIG01SRVWCRSOL1:18983/solr: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 Expired key request timestamp, elapsed=11518</title> </head> <body><h2>HTTP ERROR 401 Expired key request timestamp, elapsed=11518</h2> <table> <tr><th>URI:</th><td>/solr/admin/metrics</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Expired key request timestamp, elapsed=11518</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html> at java.base/java.util.concurrent.FutureTask.report(Unknown Source) java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://GIG01SRVWCRSOL1:18983/solr: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 Expired key request timestamp, elapsed=11518</title> </head> <body><h2>HTTP ERROR 401 Expired key request timestamp, elapsed=11518</h2> <table> <tr><th>URI:</th><td>/solr/admin/metrics</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Expired key request timestamp, elapsed=11518</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> -----Original Message----- From: Joe Jones (DHCW - Software Development) <joe.jo...@wales.nhs.uk.INVALID> Sent: 04 February 2025 15:50 To: users@solr.apache.org Subject: Upgrade issues 9.1 to 9.7 WARNING: This email originated from outside of NHS Wales. Do not open links or attachments unless you know the content is safe. Hello all, I attempted a rolling upgrade of our Solr Cloud Windows based installation today from version 9.1 to 9.7. We have 12 shards split across 2 data centers with 3 servers in each. After the first upgrade I noticed the admin GUI was not responding properly, mostly a blank page with a spinning wheel in the corner. The query engine underneath appeared fine but the admin panels were slow at best to respond. The errors seen were around "PKIAuthenticationPlugin": Expired key request timestamp Exception trying to get public key from: (http://SERVER NAME HERE) Key is null when attempting to validate signature Could not validate PKI header We have basic authentication enabled. I rolled back the upgrade on the server and still see issues with responsiveness for the nodes (restarted the vm a couple of times). Other nodes reporting "Timeout when fetching result from node " against the server the upgrade was attempted on. We did upgrade a small SIT environment but that was from 9.2 to 9.7. Any advice on how to clear this residual issue out and what can be done for a future upgrade? Thanks. Joe. Rydym yn croesawu derbyn gohebiaeth yn Gymraeg. Byddwn yn ateb y fath ohebiaeth yn Gymraeg ac ni fydd hyn yn arwain at oedi. We welcome receiving correspondence in Welsh. We will reply to such correspondence in Welsh and this will not lead to a delay.
