Solr.log is not getting created

[Reej Nayagam]

Hi All,

We have created a powershell script to run solr service in windows.
When running the service from the command line, the solr.log is created at
the configured path  - SOLR_LOGS_DIR in solr.in.cmd file
But when starting as a service, even if there is a solr.log file, it gets
renamed to solr.log.1 but new file is not created.
The other logs solr-gc.log0.current and solr-3983-console.log are created.

Any advice on what might be the problem?

*Thanks,*
*Reej*

Re: Solr.log is not getting created

[Jan Høydahl]

Check the error log from the PS script, or stdout log from Solr. Probably the 
environment used by PS script is different from running it locally. We need 
some more info on your exact setup in order to guess.

Jan

> 4. jan. 2022 kl. 12:42 skrev Reej Nayagam :
> 
> Hi All,
> 
> We have created a powershell script to run solr service in windows.
> When running the service from the command line, the solr.log is created at
> the configured path  - SOLR_LOGS_DIR in solr.in.cmd file
> But when starting as a service, even if there is a solr.log file, it gets
> renamed to solr.log.1 but new file is not created.
> The other logs solr-gc.log0.current and solr-3983-console.log are created.
> 
> Any advice on what might be the problem?
> 
> *Thanks,*
> *Reej*

Re: Non equi-joins with streaming expressions

[Damiano Albani]

Hi,

Just a quick note to mention that I've managed to implement what I wanted
in terms of non equi-joins.
Should someone be interested, I've put my code on
https://github.com/dalbani/solr-streaming-expressions.

By the way, I happened to need a startsWith function and I implemented it
quite easily.
But I'm wondering if a very generic -- if not possibly not very safe --
java() evaluator could be built.
That would open streaming expressions to the whole Java API instead of
having to write individual evaluators.
For the example of startsWith, it could look like something in the range of:

> java(val(Hello), val(World), "arg0.startsWith(arg1)")

Using say, https://www.javassist.org/, to turn the code argument into
bytecode.
What do you think?

Regards,

On Wed, Dec 29, 2021 at 12:39 PM Damiano Albani 
wrote:

> Hello,
>
> I'm new to streaming expressions, so I'm trying to understand their
> features and limitations.
> In particular the so-called "stream operators" implementing join
> operations.
> Like "innerJoin", "leftOuterJoin", etc.
>
> I see that they support a "on" parameter, defining the *equality* check
> to be performed.
> But, coming from the SQL world, I'm used to being able to use a variety of
> comparison operators in join predicates. That is, not only equality, as in
> "equi-joins".
>
> Is there a reason why the current implementation of Solr supports
> equi-joins only? Would it be technically possible (and desired) to support
> other comparison operators with joins?
> And maybe somehow allow the use of the available stream evaluators
> ?
>
> To give the context of my question: I'm trying to join 2 sets of documents
> with a hierarchical relationship.
> My goal is to join them using a "path" field on one side and
> "descendent_path" field on the other side.
> But it looks like that only doc values are accessible (and not analyzed
> ones) in streams, so I suppose I'd be left with a join criteria like this
> pseudo-code:
>
>>   on="starts_with(right.path, left.path)"
>
> Where, in this hypothetical example:
>
>>   left.path=/categories/category1"
>>   right.path=/categories/category1/sub-categories/sub-category-a"
>
>
> Or do I completely misunderstand how Solr (streams) work? ;-)
> Thanks for your help!
>
> Regards,
>
> --
> Damiano Albani
>


-- 
Damiano Albani

Re: Non equi-joins with streaming expressions

[Eric Pugh]

That looks great!  I love how (relatively) simple it all is to write your own 
logic.

One of the reasons that we added packages (bin/solr package) to Solr is so that 
if someone wants to add something like a java() evaluator, they can!

> On Jan 4, 2022, at 11:40 AM, Damiano Albani  wrote:
> 
> Hi,
> 
> Just a quick note to mention that I've managed to implement what I wanted
> in terms of non equi-joins.
> Should someone be interested, I've put my code on
> https://github.com/dalbani/solr-streaming-expressions.
> 
> By the way, I happened to need a startsWith function and I implemented it
> quite easily.
> But I'm wondering if a very generic -- if not possibly not very safe --
> java() evaluator could be built.
> That would open streaming expressions to the whole Java API instead of
> having to write individual evaluators.
> For the example of startsWith, it could look like something in the range of:
> 
>> java(val(Hello), val(World), "arg0.startsWith(arg1)")
> 
> Using say, https://www.javassist.org/, to turn the code argument into
> bytecode.
> What do you think?
> 
> Regards,
> 
> On Wed, Dec 29, 2021 at 12:39 PM Damiano Albani 
> wrote:
> 
>> Hello,
>> 
>> I'm new to streaming expressions, so I'm trying to understand their
>> features and limitations.
>> In particular the so-called "stream operators" implementing join
>> operations.
>> Like "innerJoin", "leftOuterJoin", etc.
>> 
>> I see that they support a "on" parameter, defining the *equality* check
>> to be performed.
>> But, coming from the SQL world, I'm used to being able to use a variety of
>> comparison operators in join predicates. That is, not only equality, as in
>> "equi-joins".
>> 
>> Is there a reason why the current implementation of Solr supports
>> equi-joins only? Would it be technically possible (and desired) to support
>> other comparison operators with joins?
>> And maybe somehow allow the use of the available stream evaluators
>> ?
>> 
>> To give the context of my question: I'm trying to join 2 sets of documents
>> with a hierarchical relationship.
>> My goal is to join them using a "path" field on one side and
>> "descendent_path" field on the other side.
>> But it looks like that only doc values are accessible (and not analyzed
>> ones) in streams, so I suppose I'd be left with a join criteria like this
>> pseudo-code:
>> 
>>>  on="starts_with(right.path, left.path)"
>> 
>> Where, in this hypothetical example:
>> 
>>>  left.path=/categories/category1"
>>>  right.path=/categories/category1/sub-categories/sub-category-a"
>> 
>> 
>> Or do I completely misunderstand how Solr (streams) work? ;-)
>> Thanks for your help!
>> 
>> Regards,
>> 
>> --
>> Damiano Albani
>> 
> 
> 
> -- 
> Damiano Albani

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Re: Non equi-joins with streaming expressions

[David Smiley]

I'd prefer to use Lucene's "expressions" module and thus do JavaScript.
This is more accessible to a wider audience, and I believe makes
safety/security easier (though I have not checked).

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Jan 4, 2022 at 12:30 PM Eric Pugh 
wrote:

> That looks great!  I love how (relatively) simple it all is to write your
> own logic.
>
> One of the reasons that we added packages (bin/solr package) to Solr is so
> that if someone wants to add something like a java() evaluator, they can!
>
> > On Jan 4, 2022, at 11:40 AM, Damiano Albani 
> wrote:
> >
> > Hi,
> >
> > Just a quick note to mention that I've managed to implement what I wanted
> > in terms of non equi-joins.
> > Should someone be interested, I've put my code on
> > https://github.com/dalbani/solr-streaming-expressions.
> >
> > By the way, I happened to need a startsWith function and I implemented it
> > quite easily.
> > But I'm wondering if a very generic -- if not possibly not very safe --
> > java() evaluator could be built.
> > That would open streaming expressions to the whole Java API instead of
> > having to write individual evaluators.
> > For the example of startsWith, it could look like something in the range
> of:
> >
> >> java(val(Hello), val(World), "arg0.startsWith(arg1)")
> >
> > Using say, https://www.javassist.org/, to turn the code argument into
> > bytecode.
> > What do you think?
> >
> > Regards,
> >
> > On Wed, Dec 29, 2021 at 12:39 PM Damiano Albani <
> damiano.alb...@gmail.com>
> > wrote:
> >
> >> Hello,
> >>
> >> I'm new to streaming expressions, so I'm trying to understand their
> >> features and limitations.
> >> In particular the so-called "stream operators" implementing join
> >> operations.
> >> Like "innerJoin", "leftOuterJoin", etc.
> >>
> >> I see that they support a "on" parameter, defining the *equality* check
> >> to be performed.
> >> But, coming from the SQL world, I'm used to being able to use a variety
> of
> >> comparison operators in join predicates. That is, not only equality, as
> in
> >> "equi-joins".
> >>
> >> Is there a reason why the current implementation of Solr supports
> >> equi-joins only? Would it be technically possible (and desired) to
> support
> >> other comparison operators with joins?
> >> And maybe somehow allow the use of the available stream evaluators
> >> ?
> >>
> >> To give the context of my question: I'm trying to join 2 sets of
> documents
> >> with a hierarchical relationship.
> >> My goal is to join them using a "path" field on one side and
> >> "descendent_path" field on the other side.
> >> But it looks like that only doc values are accessible (and not analyzed
> >> ones) in streams, so I suppose I'd be left with a join criteria like
> this
> >> pseudo-code:
> >>
> >>>  on="starts_with(right.path, left.path)"
> >>
> >> Where, in this hypothetical example:
> >>
> >>>  left.path=/categories/category1"
> >>>  right.path=/categories/category1/sub-categories/sub-category-a"
> >>
> >>
> >> Or do I completely misunderstand how Solr (streams) work? ;-)
> >> Thanks for your help!
> >>
> >> Regards,
> >>
> >> --
> >> Damiano Albani
> >>
> >
> >
> > --
> > Damiano Albani
>
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com <
> http://www.opensourceconnections.com/> | My Free/Busy <
> http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>

Re: Solr 6 Replication - Trouble Shooting

[mtn search]

Hi Shawn,

Thanks for your reply!  We are running Solr 6.4.3 and run multiple
instances of standalone Solr on a server modifying the port for each.  We
run many such servers.

*solrconfig.xml*
 


commit
startup


${enable.slave:false}
http://${MASTER_CORE_URL}
00:00:60



We have used the Replicate Now button on the Slave's Solr Admin UI.

Most of the time replication is working for the many cores we host.  But
enough are failing and impacting our clients that it has moved up in
priority to take a proactive approach.

Currently we restart Solr or rebuild the core to resolve the replication
issue.  However, I am thinking of trying a core reload, to see if it will
reset the replication state information.  That way I can create a script to
detect the replication error and attempt to resolve it without a Solr
restart.

Any further feedback is welcome.

Thanks,
Matt

On Mon, Jan 3, 2022 at 3:19 PM Shawn Heisey  wrote:

> On 1/3/2022 2:21 PM, mtn search wrote:
> > From the Solr Admin UI
> (server:8080/solr/old.html#/collection1/replication)
> > I can select the Replicate Now button. This will successfully replicate
> > once, but not restart the periodic replication.
> >
> > Any tips?
> >
> > I think if there was an api to modify the next replication date, that
> would
> > do the trick.
>
> Are you using the scripts to start Solr with the included jetty, or have
> you moved the webapp into a different container?  I only ask because
> port 8080 is the most commonly used port for standalone container
> installations, so I usually expect Solr on a different port number.
>
> Can you show us the replication configs on the master and the slave?
> Was the "Replicate Now" button found on the admin UI for the slave or
> for the master?  What is the exact version of Solr 6 that you are using?
>
> (In case you're wondering, I am aware that the terms in the previous
> paragraph are sensitive.  I am only using them because those are the
> terms used in the replication configuration for your Solr version.  If I
> had a reasonable way to avoid using those terms for this discussion, I
> would.)
>
> It's been many years since I last used replication in my own setups, but
> the config is fairly straightforward, and hasn't changed a whole lot
> since Solr 1.4.1, which was the last time I used it.
>
> Thanks,
> Shawn
>

Re: Non equi-joins with streaming expressions

[Damiano Albani]

Hello,

It's the first time that I hear about those Lucene expressions written in
JavaScript. Good to learn about it!
I suppose you're referring to
https://lucene.apache.org/core/9_0_0/expressions/org/apache/lucene/expressions/js/package-summary.html
?
I couldn't find much information about how to use it, especially in
combination with Solr. If someone knowledgeable could chime in, that would
be great.
Though what I see on the API documentation page at first impression, is
that the list of supported functions is pretty limited.
Actually, I think that Solr's decorators provide a similar coverage of
functions out of the box:
https://solr.apache.org/guide/8_11/stream-evaluator-reference.html.
If I can find some time, I will play with my java() decorator idea and see
if it is any good.
Especially in terms of performance, where JavaScript-in-Lucene could have
the upper hand indeed.

Regards,

On Tue, Jan 4, 2022 at 6:41 PM David Smiley  wrote:

> I'd prefer to use Lucene's "expressions" module and thus do JavaScript.
> This is more accessible to a wider audience, and I believe makes
> safety/security easier (though I have not checked).
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Jan 4, 2022 at 12:30 PM Eric Pugh  >
> wrote:
>
> > That looks great!  I love how (relatively) simple it all is to write your
> > own logic.
> >
> > One of the reasons that we added packages (bin/solr package) to Solr is
> so
> > that if someone wants to add something like a java() evaluator, they can!
> >
> > > On Jan 4, 2022, at 11:40 AM, Damiano Albani 
> > wrote:
> > >
> > > Hi,
> > >
> > > Just a quick note to mention that I've managed to implement what I
> wanted
> > > in terms of non equi-joins.
> > > Should someone be interested, I've put my code on
> > > https://github.com/dalbani/solr-streaming-expressions.
> > >
> > > By the way, I happened to need a startsWith function and I implemented
> it
> > > quite easily.
> > > But I'm wondering if a very generic -- if not possibly not very safe --
> > > java() evaluator could be built.
> > > That would open streaming expressions to the whole Java API instead of
> > > having to write individual evaluators.
> > > For the example of startsWith, it could look like something in the
> range
> > of:
> > >
> > >> java(val(Hello), val(World), "arg0.startsWith(arg1)")
> > >
> > > Using say, https://www.javassist.org/, to turn the code argument into
> > > bytecode.
> > > What do you think?
> > >
> > > Regards,
> > >
> > > On Wed, Dec 29, 2021 at 12:39 PM Damiano Albani <
> > damiano.alb...@gmail.com>
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> I'm new to streaming expressions, so I'm trying to understand their
> > >> features and limitations.
> > >> In particular the so-called "stream operators" implementing join
> > >> operations.
> > >> Like "innerJoin", "leftOuterJoin", etc.
> > >>
> > >> I see that they support a "on" parameter, defining the *equality*
> check
> > >> to be performed.
> > >> But, coming from the SQL world, I'm used to being able to use a
> variety
> > of
> > >> comparison operators in join predicates. That is, not only equality,
> as
> > in
> > >> "equi-joins".
> > >>
> > >> Is there a reason why the current implementation of Solr supports
> > >> equi-joins only? Would it be technically possible (and desired) to
> > support
> > >> other comparison operators with joins?
> > >> And maybe somehow allow the use of the available stream evaluators
> > >> ?
> > >>
> > >> To give the context of my question: I'm trying to join 2 sets of
> > documents
> > >> with a hierarchical relationship.
> > >> My goal is to join them using a "path" field on one side and
> > >> "descendent_path" field on the other side.
> > >> But it looks like that only doc values are accessible (and not
> analyzed
> > >> ones) in streams, so I suppose I'd be left with a join criteria like
> > this
> > >> pseudo-code:
> > >>
> > >>>  on="starts_with(right.path, left.path)"
> > >>
> > >> Where, in this hypothetical example:
> > >>
> > >>>  left.path=/categories/category1"
> > >>>  right.path=/categories/category1/sub-categories/sub-category-a"
> > >>
> > >>
> > >> Or do I completely misunderstand how Solr (streams) work? ;-)
> > >> Thanks for your help!
> > >>
> > >> Regards,
> > >>
> > >> --
> > >> Damiano Albani
> > >>
> > >
> > >
> > > --
> > > Damiano Albani
> >
> > ___
> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> > http://www.opensourceconnections.com <
> > http://www.opensourceconnections.com/> | My Free/Busy <
> > http://tinyurl.com/eric-cal>
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
> >
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw
> >
> >
> > This e-mail and all contents, including attachments, is considered to be
> > Company 

Re: Solr 6 Replication - Trouble Shooting

[Shawn Heisey]

On 1/4/22 2:15 PM, mtn search wrote:

Currently we restart Solr or rebuild the core to resolve the replication
issue.  However, I am thinking of trying a core reload, to see if it will
reset the replication state information.  That way I can create a script to
detect the replication error and attempt to resolve it without a Solr
restart.


Hopefully a core reload does help.  If a restart works, I would think a 
reload would too.


Are there any messages in solr.log that look like they are related to 
replication when it stops replicating?  I wonder if somehow polling gets 
disabled on the slave.  You could try the API command to enable polling:


http:///slave_host:port//solr//core_name//replication?command=enablepoll

This is documented in the ref gude:

https://solr.apache.org/guide/6_6/index-replication.html#IndexReplication-HTTPAPICommandsfortheReplicationHandler

I am guessing that you pass in the "enable.slave" and "MASTER_CORE_URL" 
properties in the slave startup?  I can't imagine that it would work at 
all if those are not defined.


Thanks,
Shawn

Re: Non equi-joins with streaming expressions

[Dennis Gove]

My recollection from working on this code years ago is that other
definitions of "equal" can be supported by creating new implementations of
the Equalitor class (
https://github.com/apache/solr/blob/main/solr/solrj/src/java/org/apache/solr/client/solrj/io/eq/Equalitor.java#L27-L30).
The purpose of the Equalitor class is not so much to say "these two values
are the same" but instead "these values can be joined on". Joins were one
of the first streaming expressions created and as such existed before
evaluators. The Equalitor class is a bit of an unfortunate holdover from
that initial implementation. Were I doing it again now I'd use evaluators
instead.

That said, it may be possible to refactor the Equalitor class as a type of
Evaluator. An approach like that would, I think, clean up what's become a
confusing holdover of that original implementation and simultaneously make
it possible to use any evaluator within a join clause.

Alternatively, it'd be possible to enhance the join classes to support
either Equalitors or Evaluators. Equalitors are constructed with this
method -
https://github.com/apache/solr/blob/main/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java#L352
- so you could enhance any place that's called from to also support
Evaluators.

Cheers,
Dennis

On Tue, Jan 4, 2022 at 5:00 PM Damiano Albani 
wrote:

> Hello,
>
> It's the first time that I hear about those Lucene expressions written in
> JavaScript. Good to learn about it!
> I suppose you're referring to
>
> https://lucene.apache.org/core/9_0_0/expressions/org/apache/lucene/expressions/js/package-summary.html
> ?
> I couldn't find much information about how to use it, especially in
> combination with Solr. If someone knowledgeable could chime in, that would
> be great.
> Though what I see on the API documentation page at first impression, is
> that the list of supported functions is pretty limited.
> Actually, I think that Solr's decorators provide a similar coverage of
> functions out of the box:
> https://solr.apache.org/guide/8_11/stream-evaluator-reference.html.
> If I can find some time, I will play with my java() decorator idea and see
> if it is any good.
> Especially in terms of performance, where JavaScript-in-Lucene could have
> the upper hand indeed.
>
> Regards,
>
> On Tue, Jan 4, 2022 at 6:41 PM David Smiley  wrote:
>
> > I'd prefer to use Lucene's "expressions" module and thus do JavaScript.
> > This is more accessible to a wider audience, and I believe makes
> > safety/security easier (though I have not checked).
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Tue, Jan 4, 2022 at 12:30 PM Eric Pugh <
> ep...@opensourceconnections.com
> > >
> > wrote:
> >
> > > That looks great!  I love how (relatively) simple it all is to write
> your
> > > own logic.
> > >
> > > One of the reasons that we added packages (bin/solr package) to Solr is
> > so
> > > that if someone wants to add something like a java() evaluator, they
> can!
> > >
> > > > On Jan 4, 2022, at 11:40 AM, Damiano Albani <
> damiano.alb...@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > Just a quick note to mention that I've managed to implement what I
> > wanted
> > > > in terms of non equi-joins.
> > > > Should someone be interested, I've put my code on
> > > > https://github.com/dalbani/solr-streaming-expressions.
> > > >
> > > > By the way, I happened to need a startsWith function and I
> implemented
> > it
> > > > quite easily.
> > > > But I'm wondering if a very generic -- if not possibly not very safe
> --
> > > > java() evaluator could be built.
> > > > That would open streaming expressions to the whole Java API instead
> of
> > > > having to write individual evaluators.
> > > > For the example of startsWith, it could look like something in the
> > range
> > > of:
> > > >
> > > >> java(val(Hello), val(World), "arg0.startsWith(arg1)")
> > > >
> > > > Using say, https://www.javassist.org/, to turn the code argument
> into
> > > > bytecode.
> > > > What do you think?
> > > >
> > > > Regards,
> > > >
> > > > On Wed, Dec 29, 2021 at 12:39 PM Damiano Albani <
> > > damiano.alb...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I'm new to streaming expressions, so I'm trying to understand their
> > > >> features and limitations.
> > > >> In particular the so-called "stream operators" implementing join
> > > >> operations.
> > > >> Like "innerJoin", "leftOuterJoin", etc.
> > > >>
> > > >> I see that they support a "on" parameter, defining the *equality*
> > check
> > > >> to be performed.
> > > >> But, coming from the SQL world, I'm used to being able to use a
> > variety
> > > of
> > > >> comparison operators in join predicates. That is, not only equality,
> > as
> > > in
> > > >> "equi-joins".
> > > >>
> > > >> Is there a reason why the current implementation of Solr supports
> > > >> equi-joins only? Would it be technically possi