hl.method=original

In Solr 9.2.1, is there a way to tweak solrconfig.xml so that hl.method=original is (once again) the default?

Re: hl.method=original

Wouldn’t using an invariant definition work? https://solr.apache.org/guide/solr/latest/configuration-guide/requesthandlers-searchcomponents.html#invariants > On Nov 11, 2023, at 12:18 PM, Oakley, Craig (NIH/NLM/NCBI) [C] > wrote: > > In Solr 9.2.1, is there a way to tweak solrconfig.xml so

How to do fastest loading and indexing

Hi, I have a CVS file with 200 fields and 100 million rows of historical and latest data. The current processing is taking 20+ hours. The schema is liked: ... ... In terms of hardware, I have 3 identical servers. One of them is used to load this CSV to create a core. What is the fastest

Re: How to do fastest loading and indexing

Are you using copy from? On Sat, Nov 11, 2023, 2:33 PM Vince McMahon wrote: > Hi, > > I have a CVS file with 200 fields and 100 million rows of historical and > latest data. > > The current processing is taking 20+ hours. > > The schema is liked: > > ... > > > > ... > > > > In terms of har

Re: How to do fastest loading and indexing

I'm not querying with catch_all at the moment, but, other developers may. I am new. Mind sharing how it matters, esp. How it makes loading n idx fast? On Sat, Nov 11, 2023, 3:05 PM Benedict Holland wrote: > Are you using copy from? > > On Sat, Nov 11, 2023, 2:33 PM Vince McMahon > > wrote:

Re: How to do fastest loading and indexing

It really depends on how you are loafing data. If you are going line by line then it's going to be very slow. Upi shpuld load datasets like this with a copy from. If you have any issues with your csv file though, it's going to be a problem. Things like commas without quotes tend to make up the most

Re: How to do fastest loading and indexing

Oh, also, this matters because copy from is a batch job that will stream your data into the table. It's extremely fast. Your indexes are not the problem. They are extremely efficient. The problem is likely how you are loading the data. So actually, how are you loading the data? Thanks, Ben On Sa

Re: How to do fastest loading and indexing

On 11/11/2023 12:32, Vince McMahon wrote: What is the fastest way to load and index this large and wide CSV file? It is taking too long, 20+ hours, now. I am assuming here that you are sending the CSV data directly to Solr and letting Solr parse it into documents. If that is incorrect, pleas

Re: How to do fastest loading and indexing

>From the Solr UI, how can I tell the number of threads are set for indexing ? On Sat, Nov 11, 2023, 5:31 PM Shawn Heisey wrote: > On 11/11/2023 12:32, Vince McMahon wrote: > > What is the fastest way to load and index this large and wide CSV file? > It > > is taking too long, 20+ hours, now. >

Re: How to do fastest loading and indexing

Benedict, Thanks for your replies. I am trying to load to Solr Core and index there, not postgres. Would you happen to know the fastest way to load and index Solr Core? Thanks. On Sat, Nov 11, 2023, 4:41 PM Benedict Holland wrote: > Oh, also, this matters because copy from is a batch job th

Re: How to do fastest loading and indexing

Shawn, Thanks for helping me out. Solr documentation has a lot of bells and whistles and I am overwhelmed. The total number of documents is 200 millions. Each line of the csv will be a document. There are 200 million lines. I have the 2 options on load-n-index The current way of getting dat

is it in mill-sec or seconds

Hi, I would like to find out the unit of the QUERY./dataImport.totalTime. Is it in mill-seconds or seconds? [image: image.png]