Thanks: I was forgetting the fact that (in order to get backups in the format
we expected) I added "incremental=false" to our backup script when we upgraded
from Solr8.5 to Solr8.11.
It trying to test this functionality, however, I now find that backups do not
seem to be working for me *at all* in Solr9.2, with or without
"incremental=false"
I keep getting a stack trace such as:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
<int name="status">500</int>
<int name="QTime">9</int>
</lst>
<lst name="error">
<str name="msg">access denied ("java.io.FilePermission"
"/netmnt/[redacted-directory]" "read")</str>
<str name="trace">java.security.AccessControlException: access denied
("java.io.FilePermission" "/netmnt/[redacted-directory]" "read")
at
java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at
java.base/java.security.AccessController.checkPermission(AccessController.java:897)
at
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at
java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661)
at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
at
java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
at java.base/java.nio.file.Files.exists(Files.java:2435)
at
org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110)
at
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404)
at
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878)
at
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548)
at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252)
at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220)
at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.base/java.lang.Thread.run(Thread.java:829)
</str>
<int name="code">500</int>
</lst>
</response>
It should be noted that the Linux account which runs Solr has read and write
access to the specified location, which is a subdirectory of what is listed in
allowPaths in all the solr.xml files of this SolrCloud, and which is mounted on
all hosts of the SolrCloud.
I have tried both V1 API and V2 API as found at
solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup
Not being familiar with V2 API, I tried various permutations of
"file:///netmnt/" (various numbers of slashes, etc): they all seem to get the
same stack trace
Does anyone have any example of a successful backup from Solr9.2?
-----Original Message-----
From: Dominique Bejean <[email protected]>
Sent: Friday, August 11, 2023 9:32 AM
To: [email protected]
Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower replication
Hi Craig,
Yes, starting with the 8.9 and 9.0 versions, Collection API allows
incremental backup and much more (corruption check, backup to Amazon S3 or
Google Cloud Storage) .
Take a look at this umbrella JIRA
https://issues.apache.org/jira/browse/SOLR-15086
Regards
Dominique
Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C]
<[email protected]> a écrit :
> RE: a better option is to use the backup/restore functionality in the
> Collections API.
>
> My impression was that there is no facility for _incremental_ backup and
> retore in the collections API: is there? To do backup and scp and restore
> of terabytes of data every few minutes does not sound practical.
>
> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL as
> well as Solr) is redundancy (for failover) in our main Data Center, and
> redundancy in read-only reasonably-concurrent copies in our second Data
> Center. For Solr8.11 (and earlier), we have had SolrCloud for redundancy in
> our main Data Center, and Leader/Follower replication to the read-only
> SolrClouds in the second Data Center. At one time, we were hoping that CDCR
> would work better: but we never managed to get CDCR to work reliably (and
> perhaps others also found it unreliable, leading to it being deprecated
> rather than being fixed). We have found that SolrCloud does not work
> reliably when spread across Data Centers, so Leader/Follower replication
> (formerly known as Master/Slave replication) has been the only way we have
> found to keep our (read-only) copies in the second Data Center only a few
> minutes behind the data in the main Data Center (the only way to provide
> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres and
> MySQL). My supervisor was asking for clarification whether you are implying
> that Leader/Follower replication is being deprecated.
>
>
> In my continuing attempts to resolve these issues, I have come across a
> related question. The error message about SolrAuthV2 prompted me to wonder
> about the V1 and V2 syntax options (such as shown at
> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload);
> and I was wondering whether Leader/Follower replication changed from using
> syntax V1 to using syntax V2, and if that might contribute to the breaking
> of permissions. As an experiment in our test environment, I setup a
> permission in security.json to allow RELOAD collection without a password.
> After confirming that my V2 syntax does work _with_ a password, I then
> attempted RELOAD collection without a password using both syntax V1 and
> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several
> permutations in security.json to allow RELOAD without password in syntax
> V2, but have not yet found a successful permutation. Are there any
> clarifications what security.json changes are needed for syntax V2? Can it
> be confirmed whether Leader/Follower replication is using V2 (in other
> words, whether that may be contributing to the permission problem)?
>
>
> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat
> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type:
> application/json' -d '{"reload":{}}'
> {
> "responseHeader":{
> "status":0,
> "QTime":255},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":219}}}}
> [11:51 dbh19850s 1152]$ curl
> "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs";
>
> {
> "responseHeader":{
> "status":0,
> "QTime":237},
> "success":{
> "nosqltest21.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":176}},
> "nosqltest22.be-md:9852_solr":{
> "responseHeader":{
> "status":0,
> "QTime":216}}}}
> [11:51 dbh19850s 1153]$ curl -X POST
> "http://localhost:$p/api/collections/helpdocs";
> -H 'Content-Type: application/json' -d '{"reload":{}}'
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 401 Authentication failed, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2>
> <table>
> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr>
> <tr><th>STATUS:</th><td>401</td></tr>
> <tr><th>MESSAGE:</th><td>Authentication failed, Response code:
> 401</td></tr>
> <tr><th>SERVLET:</th><td>default</td></tr>
> </table>
>
> </body>
> </html>
> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3
>
> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs
> HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET
> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280
> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs
> HTTP/1.1" 401 491
> [11:51 dbh19850s 1155]$ less solr.log
> ...
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] to govern resource
> [/admin/collections]
> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ]
> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"openreload8",
> "path":"/admin/collections",
> "index":9,
> "collection":null,
> "role":null,
> "params":{
> "action":["REGEX:(?i)RELOAD"],
> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access
> ...
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs]
> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ]
> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{
> "name":"catch-all-nocollection",
> "path":"/*",
> "index":24,
> "collection":null,
> "role":"allgen"}] has role, but request principal cannot be identified;
> forbidding access
>
>
> As a side note, in our experience, the only thing that has been cluttering
> up solr.log with attempts to connect without a password has been
> Leader/Follower replication (formerly known as Master/Slave replication).
>
>
> -----Original Message-----
> From: Shawn Heisey <[email protected]>
> Sent: Saturday, August 5, 2023 4:24 PM
> To: [email protected]
> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication
>
> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> > We are having problems transitioning Leader/Follower replication to
> Solr9.2.1
> >
> > In Solr8.5 and below, what was then called Master/Slave replication had
> the annoying problem that, even though we specified httpBasicAuthUser and
> httpBasicAuthPassword, it would always attempt to connect first without a
> password before retrying with a password. This made solr.log noisy with
> lots of unnecessary login failures: but at least it worked.
>
> In general, this is how basic auth via http works. The client first
> attempts the request without any credentials, and receives a 401 response.
>
> At this point, a browser would see the 401 response and display a popup
> asking for username/password.
>
> Then on future requests to that server in the current session, the
> browser sends the supplied credentials on every request to that server.
>
> If you are supplying credentials in the replication config, it should
> NOT follow that pattern ... the credentials should be always used.
>
> > Now when we are preparing to upgrade to Solr9.2.1, we are having issues
> with the following:
> > 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ]
> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
> > 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ]
> o.a.s.h.IndexFetcher Leader at:
> http://[REDACTED]/solr/sequence2_shard1_replica_n1
> is not available.
>
> The info above is valid for Solr running in standalone mode. But those
> core names indicate that you are running in SolrCloud mode.
>
> In cloud mode, Solr handles all replication. Don't attempt to actually
> configure the replication handler in cloud mode ... Solr will handle it
> all for you, and will even automatically take care of authenticating the
> requests. You don't need to configure the replication handler at all.
>
> If you are using the replication handler as a "back door" to copy
> indexes to a separate Solr install, a better option is to use the
> backup/restore functionality in the Collections API.
>
> Thanks,
> Shawn
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and are
> confident the content is safe.
>
>
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you recognize the sender and are confident the
content is safe.
