Reload greyed out in 9.2 browser GUI

[Oakley, Craig (NIH/NLM/NCBI) [C]]

After upgrading from Solr8.11.1 to Solr9.2.1, the browser GUI (at 
hostname:port/solr/#/~collections/collection_name) has the "Reload" button 
greyed out and not functional ("Add Collection" and "Delete Collection" have 
also become greyed-out, but I am not as concerned about those). This is when 
logged in as a user which can execute 
solr/admin/collections?action=RELOAD&name=collection_name using curl; but it 
would be nice to allow the browser interface to work as well. I have not found 
any clues in the request logs, nor using SolrLogAuditLoggerPlugin, nor by 
increasing logging levels for security, or in the "Major Changes" web page. Any 
ideas?

Why does edismax resets field level wild query to *:*

[rajani m]

Hi Solr Users,

 Why does edismax query reset field exist query to *:*?  Is this
expected behavior or a bug?

 Edismax query "q": "{!edismax qf=keywords v='*'}" or in the other
form "q=*&qf=keywords.en&defType=edismax"
I expected this to translate into keywords:*

Parser debug query response is as follows
"parsedquery": "+MatchAllDocsQuery(*:*)","parsedquery_toString": "+*:*","
QParser": "ExtendedDismaxQParser",

RE: [EXTERNAL] Re: hl.method=original

[Oakley, Craig (NIH/NLM/NCBI) [C]]

That worked: thanks

-Original Message-
From: Eric Pugh 
Sent: Saturday, November 11, 2023 1:12 PM
To: users@solr.apache.org
Subject: [EXTERNAL] Re: hl.method=original

Wouldn't using an invariant definition work?  
https://solr.apache.org/guide/solr/latest/configuration-guide/requesthandlers-searchcomponents.html#invariants



> On Nov 11, 2023, at 12:18 PM, Oakley, Craig (NIH/NLM/NCBI) [C] 
>  wrote:
>
> In Solr 9.2.1, is there a way to tweak solrconfig.xml so that 
> hl.method=original is (once again) the default?

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com/  | 
My Free/Busy 
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 

This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and are confident the 
content is safe.

Re: Why does edismax resets field level wild query to *:*

[Vaijanath Rao]

As far as I can recall/remember this is an expected behavior. The value for
the query is coming from the parameter v and qf is coming from the
parameter qf which in this example is set to "*". Hence q=* and qf =
keywords.en.

--Thanks and regards
Vaijanath

On Wed, Nov 15, 2023 at 7:06 AM rajani m  wrote:

> Hi Solr Users,
>
>  Why does edismax query reset field exist query to *:*?  Is this
> expected behavior or a bug?
>
>  Edismax query "q": "{!edismax qf=keywords v='*'}" or in the other
> form "q=*&qf=keywords.en&defType=edismax"
> I expected this to translate into keywords:*
>
> Parser debug query response is as follows
> "parsedquery": "+MatchAllDocsQuery(*:*)","parsedquery_toString": "+*:*","
> QParser": "ExtendedDismaxQParser",
>


-- 
I am feeling fine, healthier and Happier, what about you

Re: Why does edismax resets field level wild query to *:*

[rajani m]

Hi Vaijanath,

qf is *not* set to star but the field name "keywords". It is the value that
is set to star
 "q": "{!edismax qf=keywords v='*'}"

On Wed, Nov 15, 2023 at 10:19 AM Vaijanath Rao  wrote:

> As far as I can recall/remember this is an expected behavior. The value for
> the query is coming from the parameter v and qf is coming from the
> parameter qf which in this example is set to "*". Hence q=* and qf =
> keywords.en.
>
> --Thanks and regards
> Vaijanath
>
> On Wed, Nov 15, 2023 at 7:06 AM rajani m  wrote:
>
> > Hi Solr Users,
> >
> >  Why does edismax query reset field exist query to *:*?  Is this
> > expected behavior or a bug?
> >
> >  Edismax query "q": "{!edismax qf=keywords v='*'}" or in the other
> > form "q=*&qf=keywords.en&defType=edismax"
> > I expected this to translate into keywords:*
> >
> > Parser debug query response is as follows
> > "parsedquery": "+MatchAllDocsQuery(*:*)","parsedquery_toString": "+*:*","
> > QParser": "ExtendedDismaxQParser",
> >
>
>
> --
> I am feeling fine, healthier and Happier, what about you
>

Re: Delete large number of documents using UI

[Shawn Heisey]

On 11/14/23 08:59, Vince McMahon wrote:

Ishan, within that UI, I have the choice to set "Commit Within".  Is there
a way to via the UI to pause all indexing operations?  And how to resume,
afterward?


That just tells Solr that within the specified number of milliseconds, a 
commit will begin.  I am not sure whether the timer starts when the 
update does, or whether the timer starts when the update finishes.


The problems that DBQ has are inherent to the way DBQ works.  I only 
understand WHAT happens, not WHY.


If you change a DBQ to a multistep process where you first run the query 
to get the list of uniqueKey values, and then use that information to 
execute DeleteById updates, then the "interfering with ongoing indexing" 
that DBQ can cause will not happen.


Thanks,
Shawn

Re: Delete large number of documents using UI

[Dmitri Maziuk]

On 11/15/23 09:54, Shawn Heisey wrote:

If you change a DBQ to a multistep process where you first run the query 
to get the list of uniqueKey values, and then use that information to 
execute DeleteById updates, then the "interfering with ongoing indexing" 
that DBQ can cause will not happen.


The general problem with that is, of course, guaranteeing that the list 
won't change between the query and the last delete call.


IJS
Dima

Re: Reload greyed out in 9.2 browser GUI

[Shawn Heisey]

On 11/15/23 08:10, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:

After upgrading from Solr8.11.1 to Solr9.2.1, the browser GUI (at hostname:port/solr/#/~collections/collection_name) has 
the "Reload" button greyed out and not functional ("Add Collection" and "Delete Collection" 
have also become greyed-out, but I am not as concerned about those). This is when logged in as a user which can execute 
solr/admin/collections?action=RELOAD&name=collection_name using curl; but it would be nice to allow the browser 
interface to work as well. I have not found any clues in the request logs, nor using SolrLogAuditLoggerPlugin, nor by 
increasing logging levels for security, or in the "Major Changes" web page. Any ideas?


After starting a 9.2.1 cloud example and examining the resulting web 
page in the browser debug console, I can see that what will cause the 
reload button to be disabled is a lack of collection edit permissions 
for the logged-in user.


Because calling the reload with curl works, the overall permission 
system is functioning as expected.


Explicitly grant the collection-admin-edit permission to the logged in 
user or their group, and it should fix the UI.


The fix for SOLR-16621, which is included in 9.2.1, should have 
eliminated this problem.  Are you absolutely sure you're running 9.2.1 
and not 9.0.x or 9.1.x?


Thanks,
Shawn

Re: Why does edismax resets field level wild query to *:*

[Chris Hostetter]

:  Why does edismax query reset field exist query to *:*?  Is this
: expected behavior or a bug?
: 
:  Edismax query "q": "{!edismax qf=keywords v='*'}" or in the other
: form "q=*&qf=keywords.en&defType=edismax"
: I expected this to translate into keywords:*

this is an intentional feature of the parser(s) because 99% of the time if 
a query string contained a bare '*' (w/o an explicit field, or used as a 
suffix on some term) people expect it to mean "match everything")...

https://issues.apache.org/jira/browse/SOLR-2996

if you had given edismax a query string of 'keywords:*' (and `keywords` is 
an allowed user field in the parser configuration) it would have treated 
it as a "field 'keywords' contains some term" query.  But because the '*' 
is bare both edismax and the lucene qparser treat it as the "match all 
docs" query regardless of what the respectic qf or df params are.


-Hoss
http://www.lucidworks.com/

RE: [EXTERNAL] Re: Reload greyed out in 9.2 browser GUI

[Oakley, Craig (NIH/NLM/NCBI) [C]]

Thank you

Indeed that worked: and placing far enough down in security.json it did not 
inadvertently grant CREATE or DELETE permissions

Thanks again

-Original Message-
From: Shawn Heisey  
Sent: Wednesday, November 15, 2023 12:48 PM
To: users@solr.apache.org
Subject: [EXTERNAL] Re: Reload greyed out in 9.2 browser GUI

On 11/15/23 08:10, Oakley, Craig (NIH/NLM/NCBI) [C] wrote:
> After upgrading from Solr8.11.1 to Solr9.2.1, the browser GUI (at 
> hostname:port/solr/#/~collections/collection_name) has the "Reload" button 
> greyed out and not functional ("Add Collection" and "Delete Collection" have 
> also become greyed-out, but I am not as concerned about those). This is when 
> logged in as a user which can execute 
> solr/admin/collections?action=RELOAD&name=collection_name using curl; but it 
> would be nice to allow the browser interface to work as well. I have not 
> found any clues in the request logs, nor using SolrLogAuditLoggerPlugin, nor 
> by increasing logging levels for security, or in the "Major Changes" web 
> page. Any ideas?

After starting a 9.2.1 cloud example and examining the resulting web
page in the browser debug console, I can see that what will cause the
reload button to be disabled is a lack of collection edit permissions
for the logged-in user.

Because calling the reload with curl works, the overall permission
system is functioning as expected.

Explicitly grant the collection-admin-edit permission to the logged in
user or their group, and it should fix the UI.

The fix for SOLR-16621, which is included in 9.2.1, should have
eliminated this problem.  Are you absolutely sure you're running 9.2.1
and not 9.0.x or 9.1.x?

Thanks,
Shawn

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and are confident the 
content is safe.

Re: Why does edismax resets field level wild query to *:*

[rajani m]

Thank you, Hoss. That answered my question.

On Wed, Nov 15, 2023, 1:08 PM Chris Hostetter 
wrote:

>
> :  Why does edismax query reset field exist query to *:*?  Is this
> : expected behavior or a bug?
> :
> :  Edismax query "q": "{!edismax qf=keywords v='*'}" or in the other
> : form "q=*&qf=keywords.en&defType=edismax"
> : I expected this to translate into keywords:*
>
> this is an intentional feature of the parser(s) because 99% of the time if
> a query string contained a bare '*' (w/o an explicit field, or used as a
> suffix on some term) people expect it to mean "match everything")...
>
> https://issues.apache.org/jira/browse/SOLR-2996
>
> if you had given edismax a query string of 'keywords:*' (and `keywords` is
> an allowed user field in the parser configuration) it would have treated
> it as a "field 'keywords' contains some term" query.  But because the '*'
> is bare both edismax and the lucene qparser treat it as the "match all
> docs" query regardless of what the respectic qf or df params are.
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Delete large number of documents using UI

[Vince McMahon]

I just love the insights from all the pro replies.

Thank you.

On Wed, Nov 15, 2023, 11:11 AM Dmitri Maziuk 
wrote:

> On 11/15/23 09:54, Shawn Heisey wrote:
>
> > If you change a DBQ to a multistep process where you first run the query
> > to get the list of uniqueKey values, and then use that information to
> > execute DeleteById updates, then the "interfering with ongoing indexing"
> > that DBQ can cause will not happen.
>
> The general problem with that is, of course, guaranteeing that the list
> won't change between the query and the last delete call.
>
> IJS
> Dima
>
>

Re: G1GC not cleaning old gen

[bilal qureshi]

Hi Shawn,

I'm running facet queries with G1GC, and Solr is crashing with OOM, not
triggering GC.
All caches are disabled in solrconfig.xml.

Thanks,
Bilal


On Fri, 10 Nov, 2023, 21:32 Shawn Heisey, 
wrote:

> On 11/10/2023 06:09, bilal qureshi wrote:
> > I'm using solr cloud 9.1.1 on solr cloud cluster start initial heap it
> > takes around 400 MB out of 12 GB. After some queries on solr heap
> increases
> > to 8 GB. And when I reload the collection it reduces to 7GB.
> > using G1GC in this case.
> >
> > Running the same testcase with ZGC, it reduces heap to less than 1 GB.
>
> In order to have this be a valid comparison, you would have to trigger a
> full GC, not just reload the collection.  Solr itself will never
> manually trigger any kind of GC, you would have to add a custom plugin
> or compile a custom version of Solr.
>
> G1GC is going to concentrate on cleaning the new generation.  In order
> for garbage to be collected in the old gen, a full GC must occur.  G1GC
> tries to NOT do full GCs unless it absolutely has to.  With 8GB out of
> 12GB in use, there is plenty of free memory without the full GC, which
> is why it does not do anything to the old gen.
>
> If allocated heap gets close enough to the max (12GB in your case), then
> G1 will do a full GC.  Which will be slow and will pause Solr until it's
> done.  On an 8GB heap, I have seen G1GC pause the application for up to
> 15 seconds when doing a full GC.
>
> I know very little about how ZGC actually functions.  I have done some
> experiments with it ... gceasy.io analysis reveals that it does a lot
> more total collections than G1, but each one is significantly faster
> which means that an individual pause is very brief.  From your results,
> it appears that ZGC does clean both generations on its incremental runs.
>
> Thanks,
> Shawn
>
>