Hi Karl,
Attaching the disk removed the old replica and the core.properties
file references (which is part of core discovery), which caused the
core to not be able to load. The down replica is a ZK reference to
that replica. DELETEREPLICA and then ADDREPLICA as mentioned above
should fix it. You may consider using a snapshot and then restoring
from it when you create a new disk.
Thanks,
Patryk
On Wed, May 7, 2025 at 10:58 PM anon anon <anonimoussech...@gmail.com> wrote:
>
> did you tried
> solr delete -c YOUR_DUPLICATA
> ? I am noob and I am just asking.
>
> Le mer. 7 mai 2025 à 09:07, Karl Stoney
> <karl.sto...@autotrader.co.uk.invalid> a écrit :
>
> > Ah no worries, thanks for the reply.
> > I already have a customer operator that watches the statefulsets and does
> > some admin type stuff, guess ill shim into there the recovery logic for now.
> >
> >
> > From: Jan Høydahl <jan....@cominvent.com>
> > Date: Tuesday, 6 May 2025 at 17:45
> > To: users@solr.apache.org <users@solr.apache.org>
> > Subject: Re: Recovery from disk failure
> >
> > Hi,
> >
> > I have seen the same happening myself, and I agree it is somewhat
> > unexpected. I believe there may be some reason beind the behavior, but
> > cannot think of any right now.
> >
> > So what would work for you right now would be to get rid of the dead
> > replica in zookeeper (try DELETEREPLICA), and then do an ADDREPLICA on the
> > new empty box, which will create a new core and start syncing.
> > Not sure if you are able to remove the replica in that state, but give it
> > a try.
> > So, until we decide to build a different default behavior in the "startup
> > on empty disk but zk says we should have collection-A-shard-1-replica-2"
> > case, the best way would be to first move all replicas away from the node
> > that will upgrade disk, and then move them back again.
> >
> > Jan
> >
> > > 6. mai 2025 kl. 13:55 skrev Karl Stoney <karl.sto...@autotrader.co.uk
> > .INVALID>:
> > >
> > > Hi,
> > > I run solr cloud on GKE; and I’m trying to move my pods to a new disk
> > type. In doing so the disk will be brand new. I’ve landed in a position
> > that I’m unsure how to recover from, where the new node is not syncing data
> > from the leader.
> > >
> > > To explain exactly what’s happening, lets say I have two nodes:
> > >
> > > *
> > > solr-0
> > > *
> > > solr-1
> > >
> > > And both are active and fully replicated.
> > > I take solr-1 down, and point it at the new disk (which is empty), and
> > bring it back up.
> > > The server starts fine, I can access solr-1 via the UI, but it never
> > recovers, in the “Cloud -> Graph” UI, I can see the shard on solr-1 is
> > down.
> > >
> > > I can see it in the “Cloud -> Nodes” GUI as up, however its collections
> > have a funny state, for example: "postcodes-006_s1r9_(down): undefined",
> > vs solr-0 which shows "postcodes-006_s1r11: 847.3Mb”.
> > >
> > > I was expecting the node to come up and see its disk was empty, and
> > resync its data from the leader, but instead it’s just sat doing, nothing….
> > >
> > > The fact I’m moving to new disks is somewhat moot, more broadly this is
> > showing me that if we lost data on a node for whatever reason, it doesn’t
> > “fix itself” - which I always (maybe blindly) assumed it would, because
> > when I bring up brand new nodes (different name) it does.
> > >
> > > Could anyone advise what I’ve done wrong here, and what the process
> > should be to get a node to resend its data entirely?
> > >
> > > This is what the API shows:
> > >
> > >
> > > shard1":{
> > >
> > > "range":"80000000-7fffffff",
> > >
> > > "replicas":{
> > >
> > > "core_node10":{
> > >
> > > "core":"postcodes-006_shard1_replica_n9",
> > >
> > >
> > "node_name":"solr-1.search-solr-next.svc.cluster.local:80_solr",
> > >
> > > "type":"NRT",
> > >
> > > "state":"down",
> > >
> > > "force_set_state":"false",
> > >
> > > "base_url":"
> > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-1.search-solr-next.svc.cluster.local%2Fsolr&data=05%7C02%7CKarl.Stoney%40autotrader.co.uk%7C320a9d3baf2d456311a208dd8cbd57f6%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C638821467062718137%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sAPOw6r%2Bpw0GnNV00bPdMTiQMNYybKlk2rgKfTZIHaw%3D&reserved=0
> > "
> > >
> > > },
> > >
> > > "core_node12":{
> > >
> > > "core":"postcodes-006_shard1_replica_n11",
> > >
> > >
> > "node_name":"solr-0.search-solr-next.svc.cluster.local:80_solr",
> > >
> > > "type":"NRT",
> > >
> > > "state":"active",
> > >
> > > "leader":"true",
> > >
> > > "force_set_state":"false",
> > >
> > > "base_url":"
> > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-0.search-solr-next.svc.cluster.local%2Fsolr&data=05%7C02%7CKarl.Stoney%40autotrader.co.uk%7C320a9d3baf2d456311a208dd8cbd57f6%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C638821467062735681%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=LqHQ7N0PhNlDjUHq52Kn4AfgzV8uvFf7A2eXC%2BLlfqY%3D&reserved=0
> > ",
> > >
> > > "property.preferredleader":"true"
> > >
> > > }
> > >
> > > },
> > >
> > > "state":"active",
> > >
> > > "health":"ORANGE"
> > >
> > > }
> > >
> > >
> > >
> > >
> > > Unless expressly stated otherwise in this email, this e-mail is sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or entity to
> > whom they are addressed. If you have received this email in error please
> > notify the sender. This email message has been swept for the presence of
> > computer viruses.
> >
> >
> >
> > Unless expressly stated otherwise in this email, this e-mail is sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or entity to
> > whom they are addressed. If you have received this email in error please
> > notify the sender. This email message has been swept for the presence of
> > computer viruses.
> >
