Re: Re-index after upgrade

2023-08-28 Thread Jan Høydahl
Ok, thanks for the clarification. Jan > 28. aug. 2023 kl. 20:43 skrev Rahul Goswami : > > Yep, that check is present in Lucene 9.x as well. It will refuse to open an > index created in 7.x. > > https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core/src/java/org/apache/lucene/in

Re: Re-index after upgrade

2023-08-28 Thread Rahul Goswami
Yep, that check is present in Lucene 9.x as well. It will refuse to open an index created in 7.x. https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java#L345 https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core

Re: Re-index after upgrade

2023-08-28 Thread Shawn Heisey
On 8/28/23 05:03, Jan Høydahl wrote: Are you sure that 9.x refuse to open an index first created in 7.x? I thought that strict policy was only needed in 8.0 due to a particular lossy data structure change, and that 9.x is more lenient? I haven't actually tried it, but I believe N-1 is enforce

Re: Re-index after upgrade

2023-08-28 Thread Jan Høydahl
Are you sure that 9.x refuse to open an index first created in 7.x? I thought that strict policy was only needed in 8.0 due to a particular lossy data structure change, and that 9.x is more lenient? Jan > 24. aug. 2023 kl. 22:38 skrev Shawn Heisey : > > Version 9 behaves the same as 8. If the

Re: Re-index after upgrade

2023-08-26 Thread Shawn Heisey
On 8/25/23 15:35, Mike Drob wrote: I’m not sure if this is a Solr cloud only thing, I don’t think it works with leader/follower mode, but there’s probably similar principles to apply. Standalone Solr does not have aliases. I simulated something like that functionality with core swapping, clear

Re: Re-index after upgrade

2023-08-25 Thread Mike Drob
One thing that makes the cutover easier would be using aliases feature in your Solr cloud. You have old data in something like collection_2022 and the alias is just collection. Then new index format with new Solr version goes to collection_2023 assuming you have the spare space and CPU. Do some te

Re: Re-index after upgrade

2023-08-25 Thread Christopher Schultz
Shawn, On 8/25/23 13:04, Shawn Heisey wrote: On 8/25/23 10:44, Shawn Heisey wrote: On 8/25/23 10:21, Christopher Schultz wrote: I believe we have only used Solr 7 and above. Is there a way to confirm without actually /trying/ it? There is this very cool endpoint shared by Rahul that I did no

Re: Re-index after upgrade

2023-08-25 Thread Christopher Schultz
Tim, On 8/25/23 08:23, Tim Funk wrote: This had been my normal upgrade path now for the major upgrades: 0) Optimistically assume you have all the original data for re-inserts Check! Everything going into the index also goes into tables in our RDBMS. Solr is just there to make things go *much

Re: Re-index after upgrade

2023-08-25 Thread Shawn Heisey
On 8/25/23 10:44, Shawn Heisey wrote: On 8/25/23 10:21, Christopher Schultz wrote: I believe we have only used Solr 7 and above. Is there a way to confirm without actually /trying/ it? There is this very cool endpoint shared by Rahul that I did not know about: I can confirm that running this

Re: Re-index after upgrade

2023-08-25 Thread Shawn Heisey
On 8/25/23 10:21, Christopher Schultz wrote: I believe we have only used Solr 7 and above. Is there a way to confirm without actually /trying/ it? There is this very cool endpoint shared by Rahul that I did not know about: localhost:8983/solr//admin/segments Look for "createdVersionMajor" in

Re: Re-index after upgrade

2023-08-25 Thread Christopher Schultz
Shawn, On 8/24/23 16:38, Shawn Heisey wrote: On 8/24/23 14:02, Christopher Schultz wrote: I'm preparing to upgrade a standalone Solr server from 7.7.3 -> 8.11.2 and I wanted to confirm a few things. First, I'm assuming that the new server will be willing to open and modify the existing index

Re: Re-index after upgrade

2023-08-25 Thread Tim Funk
This had been my normal upgrade path now for the major upgrades: 0) Optimistically assume you have all the original data for re-inserts 1) Copy your config to fresh blank instance 2) Verify your solrconfig.xml has an update to date 3) Fix other solrconfig.xml deprecations such as replacing such a

Re: Re-index after upgrade

2023-08-24 Thread Shawn Heisey
On 8/24/23 14:57, Rahul Goswami wrote: localhost:8983/solr//admin/segments It contains this version stamp for each segment. Look for " *createdVersionMajor*" Eg: "_pp0":{ "name":"_pp0", "delCount":0, "softDelCount":0, "hasFieldUpdates":false, "sizeInBytes":39

Re: Re-index after upgrade

2023-08-24 Thread Rahul Goswami
Even if you delete all docs using deleteByQuery api and start indexing using the code in your email, index will still identify as being created in 7.x. So as Shawn mentioned, the only solution is to bring down the solr service, delete the "index" and "tlogs" folder, bring up the service again and r

Re: Re-index after upgrade

2023-08-24 Thread Shawn Heisey
On 8/24/23 14:02, Christopher Schultz wrote: I'm preparing to upgrade a standalone Solr server from 7.7.3 -> 8.11.2 and I wanted to confirm a few things. First, I'm assuming that the new server will be willing to open and modify the existing index after the upgrade, because I haven't seen any

Re-index after upgrade

2023-08-24 Thread Christopher Schultz
All, I'm preparing to upgrade a standalone Solr server from 7.7.3 -> 8.11.2 and I wanted to confirm a few things. First, I'm assuming that the new server will be willing to open and modify the existing index after the upgrade, because I haven't seen anything to suggest that wouldn't be the c

Re: Re-index after upgrade

2022-06-14 Thread Shawn Heisey
On 6/14/2022 10:22 AM, Christopher Schultz wrote: Does that mean I need to: 1. delete *:* 2. optimize 3. re-index everything Is #2 something available via the SolrJ client, or do I have to issue a REST call for that? This code should delete everything, commit, and optimize, all with a singl

Re: Re-index after upgrade

2022-06-14 Thread Christopher Schultz
Shawn, On 6/13/22 21:01, Shawn Heisey wrote: On 6/13/2022 1:19 PM, Christopher Schultz wrote: Okay. So if I do what I initially proposed: 1. delete *:* 2. re-index everything If you do this but do not optimize the index (which will happen practically instantaneously because the index will c

Re: Re-index after upgrade

2022-06-14 Thread Gus Heck
Alias switching is a very good option for cloud users, and one of the benefits of using cloud. OP is on "user managed" (also previously known as "standalone" and then briefly as "legacy mode") so this is not available. Order of preference: 1) build 100% new into new index (much better than the res

Re: Re-index after upgrade

2022-06-14 Thread Michael Conrad
In my experience, you need to index to a new collection, then use the alias command to point the connection endpoint to the new collection. Not sure how this works in non-cloud mode. We've tried reindexing to an existing collection, but, the created LUCENE version stamp doesn't get updated, an

Re: Re-index after upgrade

2022-06-13 Thread Shawn Heisey
On 6/13/2022 1:19 PM, Christopher Schultz wrote: Okay. So if I do what I initially proposed: 1. delete *:* 2. re-index everything If you do this but do not optimize the index (which will happen practically instantaneously because the index will consist of only deleted docs), then I can make

Re: Re-index after upgrade

2022-06-13 Thread Christopher Schultz
Shawn, On 6/13/22 14:40, Shawn Heisey wrote: On 6/13/2022 10:14 AM, Christopher Schultz wrote: 1. Re: regular re-indexes. I've just built this into my web application so it's literally a one-click administrative background-process kick-off. I've been trying to get automatic schema-provisionin

Re: Re-index after upgrade

2022-06-13 Thread Shawn Heisey
On 6/13/2022 10:14 AM, Christopher Schultz wrote: 1. Re: regular re-indexes. I've just built this into my web application so it's literally a one-click administrative background-process kick-off. I've been trying to get automatic schema-provisioning as well (see my recent posts to users@) just

Re: Re-index after upgrade

2022-06-13 Thread Christopher Schultz
Gus, On 6/12/22 17:57, Gus Heck wrote: What Thomas said, if possible... Definitely set up a test system if you have the resources. Building a new index from scratch ensures that nothing is lurking unconverted and allows you to move to a newer index format. One specific cost of re-indexing into

Re: Re-index after upgrade

2022-06-13 Thread Christopher Schultz
Dave, On 6/12/22 14:21, Dave wrote: You don’t need a new core/collection, just reindex everything again. Ideally since you’re using standalone (way better than cloud imo) you can use the same indexer, just do an integrity check after the fact to make sure the document counts are the same. You

Re: Re-index after upgrade

2022-06-12 Thread Gus Heck
What Thomas said, if possible... Definitely set up a test system if you have the resources. Building a new index from scratch ensures that nothing is lurking unconverted and allows you to move to a newer index format. One specific cost of re-indexing into the old index is that the index upgrader t

Re: Re-index after upgrade

2022-06-12 Thread Thomas Corthals
Or if you have the resources, set up a separate machine for the new Solr version and reindex and test against that one before switching. Op zo 12 jun. 2022 20:21 schreef Dave : > You don’t need a new core/collection, just reindex everything again. > Ideally since you’re using standalone (way bett

Re: Re-index after upgrade

2022-06-12 Thread Dave
You don’t need a new core/collection, just reindex everything again. Ideally since you’re using standalone (way better than cloud imo) you can use the same indexer, just do an integrity check after the fact to make sure the document counts are the same. You don’t really need to do that delete if

Re-index after upgrade

2022-06-12 Thread Christopher Schultz
All, We've been using the same major version of Solr for years so haven't had to do this yet, but we are preparing to upgrade between major versions, now. After upgrading, I'm assuming that the existing index is "usable" but I've read many times that "you should reindex after a major version