Hi, Have you checked that the date/time of all six servers are 100% in sync? Have you read through the Upgrade Notes for all 9.x versions to figure out if you need to take special precaution during rolling upgrade between those versions?
The PKI auth method is used for internal traffic betwen nodes, such as leg 2 of a distributed request. The nodes involved will exhange PKI key information and part of the protocol is time sensitive. It is likely that the keys change during the upgrade too, but I'm not sure how the nodes handle retrying of key exhange. In any event it is strange that even after a rollback the problems would persist. There could be recoveries going on adding load to the cluster, just a guess. Jan > 5. feb. 2025 kl. 11:09 skrev Joe Jones (DHCW - Software Development) > <joe.jo...@wales.nhs.uk.INVALID>: > > Really could do with some insight on this. > > One of the nodes on the server reports constant connection errors in and out. > > I see things like: > 2025-02-05 10:03:47.518 WARN (qtp1959239586-70) [] > o.a.s.h.a.AdminHandlersProxy Exception when fetching result from node > GIG01SRVWCRSOL1:18983_solr => java.util.concurrent.ExecutionException: > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: > Error from server at http://GIG01SRVWCRSOL1:18983/solr: Expected mime type > application/octet-stream but got text/html. <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> > <title>Error 401 Expired key request timestamp, elapsed=11518</title> > </head> > <body><h2>HTTP ERROR 401 Expired key request timestamp, elapsed=11518</h2> > <table> > <tr><th>URI:</th><td>/solr/admin/metrics</td></tr> > <tr><th>STATUS:</th><td>401</td></tr> > <tr><th>MESSAGE:</th><td>Expired key request timestamp, > elapsed=11518</td></tr> > <tr><th>SERVLET:</th><td>default</td></tr> > </table> > > </body> > </html> > > at java.base/java.util.concurrent.FutureTask.report(Unknown Source) > java.util.concurrent.ExecutionException: > org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: > Error from server at http://GIG01SRVWCRSOL1:18983/solr: Expected mime type > application/octet-stream but got text/html. <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> > <title>Error 401 Expired key request timestamp, elapsed=11518</title> > </head> > <body><h2>HTTP ERROR 401 Expired key request timestamp, elapsed=11518</h2> > <table> > <tr><th>URI:</th><td>/solr/admin/metrics</td></tr> > <tr><th>STATUS:</th><td>401</td></tr> > <tr><th>MESSAGE:</th><td>Expired key request timestamp, > elapsed=11518</td></tr> > <tr><th>SERVLET:</th><td>default</td></tr> > > -----Original Message----- > From: Joe Jones (DHCW - Software Development) > <joe.jo...@wales.nhs.uk.INVALID> > Sent: 04 February 2025 15:50 > To: users@solr.apache.org > Subject: Upgrade issues 9.1 to 9.7 > > WARNING: This email originated from outside of NHS Wales. Do not open links > or attachments unless you know the content is safe. > > > Hello all, > > I attempted a rolling upgrade of our Solr Cloud Windows based installation > today from version 9.1 to 9.7. > We have 12 shards split across 2 data centers with 3 servers in each. > > After the first upgrade I noticed the admin GUI was not responding properly, > mostly a blank page with a spinning wheel in the corner. The query engine > underneath appeared fine but the admin panels were slow at best to respond. > > The errors seen were around "PKIAuthenticationPlugin": > Expired key request timestamp > Exception trying to get public key from: (http://SERVER NAME HERE) Key is > null when attempting to validate signature Could not validate PKI header > > We have basic authentication enabled. > > I rolled back the upgrade on the server and still see issues with > responsiveness for the nodes (restarted the vm a couple of times). > Other nodes reporting "Timeout when fetching result from node " against the > server the upgrade was attempted on. > > We did upgrade a small SIT environment but that was from 9.2 to 9.7. > > Any advice on how to clear this residual issue out and what can be done for a > future upgrade? > > Thanks. > > Joe. > > Rydym yn croesawu derbyn gohebiaeth yn Gymraeg. Byddwn yn ateb y fath > ohebiaeth yn Gymraeg ac ni fydd hyn yn arwain at oedi. > We welcome receiving correspondence in Welsh. We will reply to such > correspondence in Welsh and this will not lead to a delay.
