I was not pre-creating the folder in the S3 bucket in my previous tests. When I create that `temp1` folder (without `/` at the start), the following 4 variants Houston just pasted work fine
I think the confusion (and I have been fooled again just now) is that the location=s3://temp1 does not contain the bucket name (it is just s3 prefix and the folder) On 2022/02/07 17:59:09 Houston Putman wrote: > Sorry for the back and forth everyone, I'm trying to understand the issue, > but I absolutely cannot replicate this locally. > > In the S3 console, I create a directory named "temp1/" *(Please note, this > is not the same as "/temp1/", the actual folder in the bucket CANNOT start > with a "/")* under the bucket that I am using. > When I use the following Solr APIs, they ALL work for me: > > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=temp1 > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=/temp1 > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=s3://temp1 > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=s3:///temp1 > > The only possible reason I can think of for so many people to be seeing > errors is that they are pre-creating directories in S3 that have "/" > prepended to the name. > This is already documented in the ref guide > <https://solr.apache.org/guide/8_11/making-and-restoring-backups.html#s3backuprepository>, > but we can think of verbiage to make it more clear. > > I understand that we can make it so that "location" is not a required > property and default it to "/", but that doesn't make sense for all backup > repository types, such as a fileSystem backup. > > - Houston > > On Mon, Feb 7, 2022 at 8:54 AM Michael Conrad <[email protected]> wrote: > > > The restore process allows restoring to a collection with a different > > name than the original. So I would expect the requirement to have a > > collection already existing to be legit. > > > > The documentation does say (or at least implies) that to restore the > > number of shards in the destination collection has to match the backup. > > > > On 2/5/22 12:02, Eric Charles wrote: > > > ... and thx Houston and other participants to this thread. > > > > > > I have just restored the backup, and it works fine. The only point is > > that I had to create the collection before restoring (I had dropped it). I > > wonder if I have to recreate the collection with the same number of > > shards... ? > > > > > > On 2022/02/05 16:55:26 Eric Charles wrote: > > >> I have faced similar questions to get it working. I have first opened > > an issue on the solr-operatorhttps:// > > github.com/apache/solr-operator/issues/404 and then was able to > > replicate the issue on my laptop, so came to the conclusion the issue was > > at solr level, and not solr-operator. > > >> > > >> Without Sergio reply, I would not have been able to make sense of the > > parameters and get is working. Thx! > > >> > > >> On 2022/01/27 14:01:46 Sergio García Maroto wrote: > > >>> Hi, > > >>> > > >>> Correct. Using "location=backupfolder" didn't work for me. > > >>> > > >>> The only way I made it work is with location=s3:/ > > >>> Below my sample url which works well. > > >>> > > http://servername:8983/solr/admin/collections?action=BACKUP&name=personbackup&collection=person&repository=s3&location=s3:/ > > >>> > > >>> > > >>> On Tue, 25 Jan 2022 at 17:23, Houston Putman<[email protected]> > > wrote: > > >>> > > >>>> Thanks for all of the information everybody. I want to determine if > > this is > > >>>> actually a bug before we release 9.0 > > >>>> > > >>>> First, I want to clear up the usage of the "location" parameter: > > >>>> > > >>>> - It is required, but you can provide "/" as an "empty" > > directory, much > > >>>> like "s3:/". > > >>>> - You don't have to include "s3:/" or "s3://". You can you "/dir", > > >>>> "dir", "s3:/dir" or "s3://dir". All of these options will > > eventually be > > >>>> converted to the "dir/" directory in your bucket. > > >>>> - The s3 repository does not allow for directory names starting > > with > > >>>> "/". In general this is to allow all of the above ^ examples to > > compute > > >>>> to > > >>>> the same thing without users being confused how many '/'s they > > need > > >>>> after > > >>>> "s3:". Now I see that this may have contributed to the confusion, > > but we > > >>>> can always improve the documentation in the ref-guide. > > >>>> > > >>>> Now with these points cleared up, you said that you were unable to > > use the > > >>>> collections API calls with "location=backupfolder" did not work, even > > >>>> though you had a directory in your bucket named "backupfolder". Just > > to be > > >>>> clear, in S3 the folder did not have a "/" prefix correct? If so, > > this is > > >>>> indeed a bug and something that we should fix. I'll do more thorough > > >>>> testing once you confirm that the directory is following the > > expectation. > > >>>> > > >>>> - Houston > > >>>> > > >>>> On Sat, Jan 22, 2022 at 3:32 AM Shawn Heisey<[email protected]> > > wrote: > > >>>> > > >>>>> On 1/22/2022 12:38 AM, Sergio García Maroto wrote: > > >>>>>> The only difference I can see i am using collection API trying to > > >>>> backup > > >>>>>> solr cloud collection. > > >>>>> That could do it. I have done very little with SolrCloud beyond > > >>>>> occasionally firing up the cloud example. > > >>>>> > > >>>>> The Collections API backup is most likely a different code path than > > the > > >>>>> replication handler backup. I wonder if the behavior you're seeing > > >>>>> would be considered a bug. > > >>>>> > > >>>>> Thanks, > > >>>>> Shawn > > >>>>> > > >
