Re: How to do fastest loading and indexing

2023-11-12 Thread Mikhail Khludnev
> .../200mmCsvCore/dataimport? refers to https://solr.apache.org/guide/8_6/uploading-structured-data-store-data-with-the-data-import-handler.html which was extracted into https://github.com/SearchScale/dataimporthandler It's usually slow. The symptoms of non-efficient slowness is low cpu utilizatio

Re: is it in mill-sec or seconds

2023-11-12 Thread Shawn Heisey
On 11/11/2023 19:46, Vince McMahon wrote: Hi, I would like to find out the unit of the QUERY./dataImport.totalTime. Is it in mill-seconds or seconds? image.png The image you attached did not make it. The mailing list eats attachments. All times that I have seen Solr report are in milli

Re: How to do fastest loading and indexing

2023-11-12 Thread Shawn Heisey
On 11/11/2023 17:30, Vince McMahon wrote: From the Solr UI, how can I tell the number of threads are set for indexing ? If you are not using threads (or multiple processes) in your program to send data to Solr, then the indexing is single-threaded. Thanks, Shawn

Re: is it in mill-sec or seconds

2023-11-12 Thread Vince McMahon
Thanks. On Sun, Nov 12, 2023, 9:18 AM Shawn Heisey wrote: > On 11/11/2023 19:46, Vince McMahon wrote: > > Hi, > > > > I would like to find out the unit of the QUERY./dataImport.totalTime. > > Is it in mill-seconds or seconds? > > > > > > > > image.png > > The image you attached did not make it.

Re: How to do fastest loading and indexing

2023-11-12 Thread Vince McMahon
Hi, Shawn, So, if I split the single cvs into two and using two programs sending each of the splits, Solr will handle the parallel loading with multiple threads. I don't have to make changes to Solr, right? On Sun, Nov 12, 2023, 9:19 AM Shawn Heisey wrote: > On 11/11/2023 17:30, Vince McMaho

Re: How to do fastest loading and indexing

2023-11-12 Thread Vince McMahon
Hi, Mikhai. I am very encouraged by your reply. I will split the csv into smaller ones and give this a try. https://solr.apache.org/guide/7_1/uploading-data-with-index-handlers.html#csv-formatted-index-updates Could you confirm my understanding of Solr's language? Is Solr Indexing referring to

Re: How to do fastest loading and indexing

2023-11-12 Thread Mikhail Khludnev
> > using the > following will load and will index: indeed On Sun, Nov 12, 2023 at 6:53 PM Vince McMahon wrote: > Hi, Mikhai. > > I am very encouraged by your reply. > > I will split the csv into smaller ones and give this a try. > > https://solr.apache.org/guide/7_1/uploading-data-with-index-

Re: How to do fastest loading and indexing

2023-11-12 Thread Andy Lester
> On Nov 12, 2023, at 9:16 AM, Vince McMahon > wrote: > > So, if I split the single cvs into two and using two programs sending each > of the splits, Solr will handle the parallel loading with multiple > threads. I don't have to make changes to Solr, right? Yes, that's correct. We were lo

Re: How to do fastest loading and indexing

2023-11-12 Thread ufuk yılmaz
I’d add that committing less frequently, especially not committing in every update request would speed things up, if you don’t need to search at the same time (both soft and hard commit). ~ufuk yilmaz — > On 12 Nov 2023, at 20:51, Andy Lester wrote: > > > >> On Nov 12, 2023, at 9:16 AM, Vi