Re: Suggestions to improve Star queries latencies

2023-04-20 Thread Michael Gibney 
> It is a query with popularity and recency boosts, requesting the first 100
> docs with 3 fields per doc.

It sounds like you are scoring/sorting, so the optimization that
Mikhail mentioned would not apply (your use-case is not
"sort-irrelevant"). Can you share more about specifically how your
implementing/invoking your popularity/recency boosts, and how you're
applying the "with three fields per doc" requirement?

On Wed, Apr 19, 2023 at 5:23 PM Joel Bernstein  wrote:
>
> To send the query to a single shard you can add the parameter
> "distrib=false" to the query and it will stay on that shard.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Wed, Apr 19, 2023 at 5:21 PM Joel Bernstein  wrote:
>
> > You're hunting for a bottleneck. Here is how I would go about finding it:
> >
> > First I would run the query on a single shard and see how long it takes.
> > If the single shard is slow you've found your bottleneck. If its fast then
> > try the same query on each shard, one of the shards might be slow and
> > you've found your bottleneck.
> >
> > If all the shards are fast then it would seem the bottleneck is the
> > aggregator node.
> >
> > Once you've found the bottleneck then you need to start improving the
> > throughput. Let us know what you find and then we can move on to discuss
> > how to improve the throughput at the bottleneck.
> >
> > If its very fast thats
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, Apr 19, 2023 at 3:57 PM Rajani Maski 
> > wrote:
> >
> >> Thank you, Mikhail.
> >>
> >>
> >> On Wed, Apr 19, 2023 at 7:59 AM Mikhail Khludnev  wrote:
> >>
> >> > Hello, Rajani.
> >> > I meant [SOLR-14765] optimize DocList creation by skipping sort for
> >> > sort-irrelevant cases - ASF JIRA (apache.org)
> >> > 
> >> >
> >> > On Wed, Apr 19, 2023 at 4:05 AM Rajani Maski 
> >> > wrote:
> >> >
> >> > > Hi Mikhail,
> >> > >
> >> > >Yes, 9.1.1, that should be helpful, can you please point me to the
> >> > > related jira(s) and/or docs?
> >> > >
> >> > > Thank you,
> >> > > Rajani
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Apr 17, 2023 at 2:09 AM Mikhail Khludnev 
> >> > wrote:
> >> > >
> >> > > > Hello Rajani.
> >> > > > Which version are you running? IIRC 9.1.2 has some
> >> > > > improvement about caching short queries.
> >> > > >
> >> > > > On Sun, Apr 16, 2023 at 4:25 PM Rajani Maski  >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Solr Users,
> >> > > > >
> >> > > > > What are your suggestions to improve star queries latencies? By
> >> star
> >> > > > > queries I mean "*:*" or single term queries having boost formulas
> >> > > (such
> >> > > > as
> >> > > > > doc recency and many others) taking 10 or more seconds. It is a
> >> large
> >> > > > > collection with good compute resources, however I am guessing this
> >> > may
> >> > > be
> >> > > > > because each shard has too many documents and I noticed per shard
> >> > > > response
> >> > > > > time also is high.
> >> > > > >
> >> > > > > Splitting shards could be an option however it is already an
> >> > > > > evenly distributed, composite router, 96 shards collection, I am
> >> > > > > concerned that more than 100 shards per collection can lead to
> >> > > > exhaustively
> >> > > > > searching too many shards and aggregation issues. What are your
> >> > > thoughts?
> >> > > > >
> >> > > > > Can we make use of any caches, query result cache or other
> >> caches, in
> >> > > > solr
> >> > > > > that allows warming up and persisting these queries results in
> >> ram,
> >> > and
> >> > > > > that maybe helps reduce this query time?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Rajani
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Sincerely yours
> >> > > > Mikhail Khludnev
> >> > > > https://t.me/MUST_SEARCH
> >> > > > A caveat: Cyrillic!
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Sincerely yours
> >> > Mikhail Khludnev
> >> > https://t.me/MUST_SEARCH
> >> > A caveat: Cyrillic!
> >> >
> >>
> >

[ANNOUNCE] DataImportHandler 9.0.0 released

2023-04-20 Thread Ishan Chattopadhyaya 
Hi all,

DataImportHandler 9.0.0 has now been released for Solr 9.0.0, thanks to
Krzysztof Debski [0].

To use it with the package manager, follow instructions here:
https://github.com/SearchScale/dataimporthandler#installing-and-running

To grab the binaries, please check:
https://github.com/SearchScale/dataimporthandler/tree/master/repo

Documentation is not updated to 9.0 yet, but an older version is documented
here:
https://solr.apache.org/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html

Earlier today, I had reached out to Rohit, current maintainer of
DataImportHandler (github.com/rohitbemax), regarding new releases. He
requested me to take over the ownership of the repository since his current
role doesn't allow him to make FOSS contributions without due process and
approvals. In the interest of the community, I have transferred the
repository to https://github.com/SearchScale/dataimporthandler.

Please note that DataImportHandler is no longer a first party, officially
supported module/plugin/package, but it is now a community maintained,
unofficial project. If you're using DIH in production, it would be valuable
to have you join the maintainers and keep the project alive.

Immediate call for action: upgrade DIH for Solr 9.1 and 9.2.

Thanks and regards,
--
Ishan Chattopadhyaya
* Solr Consultant, SearchScale
* Committer & PMC Member, Apache Solr


[0] - https://github.com/SearchScale/dataimporthandler/pull/33

Re: [ANNOUNCE] DataImportHandler 9.0.0 released

2023-04-20 Thread Ishan Chattopadhyaya 
A more up to date documentation link is here:
https://solr.apache.org/guide/8_11/uploading-structured-data-store-data-with-the-data-import-handler.html

On Thu, 20 Apr 2023 at 18:59, Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Hi all,
>
> DataImportHandler 9.0.0 has now been released for Solr 9.0.0, thanks to
> Krzysztof Debski [0].
>
> To use it with the package manager, follow instructions here:
> https://github.com/SearchScale/dataimporthandler#installing-and-running
>
> To grab the binaries, please check:
> https://github.com/SearchScale/dataimporthandler/tree/master/repo
>
> Documentation is not updated to 9.0 yet, but an older version is
> documented here:
> https://solr.apache.org/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html
>
> Earlier today, I had reached out to Rohit, current maintainer of
> DataImportHandler (github.com/rohitbemax), regarding new releases. He
> requested me to take over the ownership of the repository since his current
> role doesn't allow him to make FOSS contributions without due process and
> approvals. In the interest of the community, I have transferred the
> repository to https://github.com/SearchScale/dataimporthandler.
>
> Please note that DataImportHandler is no longer a first party, officially
> supported module/plugin/package, but it is now a community maintained,
> unofficial project. If you're using DIH in production, it would be valuable
> to have you join the maintainers and keep the project alive.
>
> Immediate call for action: upgrade DIH for Solr 9.1 and 9.2.
>
> Thanks and regards,
> --
> Ishan Chattopadhyaya
> * Solr Consultant, SearchScale
> * Committer & PMC Member, Apache Solr
>
>
> [0] - https://github.com/SearchScale/dataimporthandler/pull/33
>

Re: Query on Solr and ZK ports

2023-04-20 Thread Ishan Chattopadhyaya 
ZK admin server port is 8080,
https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperAdmin.html#sc_adminserver

In my usage in the past, it spits out useful metrics around number of times
a znode was accessed.

On Tue, 18 Apr, 2023, 1:50 pm Jan Høydahl,  wrote:

> Hi,
>
> See https://unix.stackexchange.com/questions/684348/solr-ports-in-use for
> an answer.
> The STOP_PORT is the Jetty Servlet container's way to shutting down. It
> will only listen on localhost, so you need to use bin/solr stop command on
> the host, i.e. no need to expose this port to the outside.
>
> Wrt ZK ports, Only port 2181 needs to be open between Solr and ZK, but
> between the Zookeepers, also ports 2888 and 3888 needs to be open for
> internal communication, see
> https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.0/bk_reference/content/zookeeper-ports.html
> I'm not aware of other ports.
>
> Jan
>
> > 18. apr. 2023 kl. 05:54 skrev HariBabu kuruva  >:
> >
> > Can someone please help with this information?
> >
> > On Tue, Apr 4, 2023 at 10:01 PM HariBabu kuruva <
> hari2708.kur...@gmail.com>
> > wrote:
> >
> >> Hi All,
> >>
> >> I could see the solr process is Listening on 7981 port along with the
> >> normal solr port(8981). It is shown as DSTOP PORT , when I grep solr
> >> process. Could you please give more details on this port, Can we disable
> >> this ?
> >>
> >> With regards to Zookeeper I could see port 8080 as a ZK admin port, How
> >> can I use this, can i disable it if I don't want it ?
> >> Also I could see ZK is listening on some random port (43801) along with
> >> the other ports. Please throw some light on this.
> >>
> >> --
> >>
> >> Thanks and Regards,
> >> Hari
> >> Mobile:9790756568
> >>
> >
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
>
>

Apache Solr Schemeless mode with .Net Core

2023-04-20 Thread Dipu Divakaran 
Team,

Can someone provide me any examples/link/document for implementing Apache Solr 
Indexing and Search with schemeless mode in .Net core?

Thanks
DD
This e-mail is intended solely for the person or entity to which it is 
addressed and may contain confidential and/or privileged information. Any 
review, dissemination, copying, printing or other use of this e-mail by persons 
or entities other than the addressee is prohibited. If you have received this 
e-mail in error, please contact the sender immediately and delete this e-mail 
and any attachments from any device.

Re: [ANNOUNCE] DataImportHandler 9.0.0 released

2023-04-20 Thread David Smiley 
Yay, finally!

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Thu, Apr 20, 2023 at 9:30 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Hi all,
>
> DataImportHandler 9.0.0 has now been released for Solr 9.0.0, thanks to
> Krzysztof Debski [0].
>
> To use it with the package manager, follow instructions here:
> https://github.com/SearchScale/dataimporthandler#installing-and-running
>
> To grab the binaries, please check:
> https://github.com/SearchScale/dataimporthandler/tree/master/repo
>
> Documentation is not updated to 9.0 yet, but an older version is documented
> here:
>
> https://solr.apache.org/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html
>
> Earlier today, I had reached out to Rohit, current maintainer of
> DataImportHandler (github.com/rohitbemax), regarding new releases. He
> requested me to take over the ownership of the repository since his current
> role doesn't allow him to make FOSS contributions without due process and
> approvals. In the interest of the community, I have transferred the
> repository to https://github.com/SearchScale/dataimporthandler.
>
> Please note that DataImportHandler is no longer a first party, officially
> supported module/plugin/package, but it is now a community maintained,
> unofficial project. If you're using DIH in production, it would be valuable
> to have you join the maintainers and keep the project alive.
>
> Immediate call for action: upgrade DIH for Solr 9.1 and 9.2.
>
> Thanks and regards,
> --
> Ishan Chattopadhyaya
> * Solr Consultant, SearchScale
> * Committer & PMC Member, Apache Solr
>
>
> [0] - https://github.com/SearchScale/dataimporthandler/pull/33
>