Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-08-06 Thread Hakan Özler
Hey Pierre, See EnvUtils class, which is now the recommended way to get system > properties or environment > variables in Solr code. > That's very useful! For some reason in your case, it seems the "cached" thread pool does not > grow as expected. If you have 5 replicas or more queued to be bac

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-08-06 Thread Pierre Salagnac
Hi Hakan, > I was also wondering if the max thread size, which is currently 5 in 9.4+, could be configurable Yes, this makes sense to have this configuration. See EnvUtils class, which is now the recommended way to get system properties or environment variables in Solr code. The behavior before

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-08-05 Thread Hakan Özler
A) < > klian...@bloomberg.net> wrote: > >> Also was this just taking a snapshot backup of the segment files or did >> this also include uploading to S3? >> >> -Kevin >> >> From: users@solr.apache.org At: 07/31/24 15:22:58 UTC-4:00To: >> user

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-08-01 Thread Hakan Özler
-Kevin > > From: users@solr.apache.org At: 07/31/24 15:22:58 UTC-4:00To: > users@solr.apache.org > Subject: Re: Significant Backup/Restore Performance Degradation for Large > Collections > > Was this change tested on a cloud that was also taking active ingest/query > re

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-07-31 Thread Kevin Liang (BLOOMBERG/ 919 3RD A)
Also was this just taking a snapshot backup of the segment files or did this also include uploading to S3? -Kevin From: users@solr.apache.org At: 07/31/24 15:22:58 UTC-4:00To: users@solr.apache.org Subject: Re: Significant Backup/Restore Performance Degradation for Large Collections Was

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-07-31 Thread Kevin Liang (BLOOMBERG/ 919 3RD A)
users@solr.apache.org Subject: Re: Significant Backup/Restore Performance Degradation for Large Collections Just a heads up, with the patch mentioned above, we managed to backup a data of 3TB in 50 minutes with `solr.maxExpensiveTaskThreads=5` [1] I would like to contribute to Solr, however, I'm

Re: Significant Backup/Restore Performance Degradation for Large Collections

2024-07-31 Thread Hakan Özler
Just a heads up, with the patch mentioned above, we managed to backup a data of 3TB in 50 minutes with `solr.maxExpensiveTaskThreads=5` [1] I would like to contribute to Solr, however, I'm unsure of the steps I should take if no one is available to take on this patch. 1. https://imgur.com/a/AAd0c

Significant Backup/Restore Performance Degradation for Large Collections

2024-07-30 Thread Hakan Özler
Hi!, We're experiencing performance issues in the recent Solr versions — 9.5.0 and 9.6.1 — regarding backup and restore. In 9.2.1, we could take a backup of 10TB data in just 1 and a half hours. Currently, as of 9.5.0, taking a backup of the collection takes 7 hours! We're unable to make use of di