Do the actual cores still exist on disk? Are you using EBS or instance-backed storage? If you're using instance-backed storage and your EC2 instance is restarted and is moved to different hardware then your instance storage is gone (it's ephemeral) so the actual indexes on disk will be gone.
When your host is rebooted, is the instance name changing? Solr uses hostnames or IPs as identifiers and if your EC2 instance is restarted it may be moved to different hardware resulting in a new instance/hostname/IP which would look like a new instance in the cluster. On Wed, Jan 26, 2022, 10:23 PM Modassar Ather <modather1...@gmail.com> wrote: > Hi, > > The Solr cluster is running on a large ec2 instance. Some more details on > the environment. > > Server : AWS-EC2 instance > Zookeeper : Standalone external zookeeper. > Solr index : On AWS - EFS > Zookeeper data directory : Pointing to a folder on ec2 instance > > The core deletion is not happening on Solr nodes restart but only on system > reboot. > Your inputs will be very helpful. > > Thanks, > Modassar > > > On Tue, Jan 25, 2022 at 12:22 PM Modassar Ather <modather1...@gmail.com> > wrote: > > > Thanks for the reply Mike. > > > > The zookeeper data directory is set to a different location and not to > > /tmp. > > > > The deletion is happening only on system reboot. > > > > Thanks, > > Modassar > > > > On Tue, Jan 25, 2022 at 1:33 AM Mike Drob <md...@mdrob.com> wrote: > > > >> Make sure you’re not using /tmp for your data directory. > >> > >> Do you see deletions on system reboot only or also process restart? > >> > >> On Mon, Jan 24, 2022 at 12:37 PM Modassar Ather <modather1...@gmail.com > > > >> wrote: > >> > >> > Hi, > >> > > >> > Please provide your suggestions to fix the issue. Is there some > >> > configuration which I am missing so the nodes are getting deleted? > >> > I tried recreating new cores whenever the Solr server restarted but > that > >> > has issues like nodes not coming up on certain exceptions. > >> > > >> > Thanks, > >> > Modassar > >> > > >> > On Thu, Jan 20, 2022 at 10:27 AM Shawn Heisey <apa...@elyograg.org> > >> wrote: > >> > > >> > > On 1/19/2022 9:37 PM, Modassar Ather wrote: > >> > > > The clusterstate.json file is empty. > >> > > > >> > > It's completely normal for recent Solr versions to have an empty > >> > > clusterstate.json file. The empty file is still used by Solr's > >> > > interaction with zookeeper, but the actual clusterstate is now kept > >> on a > >> > > per-collection basis in state.json files. I think that > >> > > clusterstate.json will no longer be used in a future Solr version. > >> > > > >> > > Thanks, > >> > > Shawn > >> > > > >> > > >> > > >
