Compacting JSON-Output

[Andreas Mock]

Hi all,

I searched around for a while now but couldn't find an answer to my problem.
We want to use the JSONWriter which presents a really nice human readable
output.

Is there a way to force this writer to compact the JSON format as good as
possible to reduce the gross size of the reply?

Best regards
Andreas Mock

Re: Compacting JSON-Output

[Nicolas Franck]

https://solr.apache.org/guide/6_6/response-writers.html#ResponseWriters-TheindentParameter

> On 15 Apr 2021, at 09:32, Andreas Mock  wrote:
> 
> Hi all,
> 
> I searched around for a while now but couldn't find an answer to my problem.
> We want to use the JSONWriter which presents a really nice human readable
> output.
> 
> Is there a way to force this writer to compact the JSON format as good as
> possible to reduce the gross size of the reply?
> 
> Best regards
> Andreas Mock
> 

Distributed search

[Norbert Bodnar]

Hi.

Hope you are doing well.

I would like to start with introducing the problem we are facing.

Our(let's call us CompanyB working on WebB) primary goal is to create a web
application, which will expand an existing web(WebA developed by CompanyA)
application's searching possibilities.

Let's call this existing web application WebA. WebA is deployed on multiple
clients and every client has its own set of documents. On bigger instances,
the index is updated almost every minute and the size of the index is
sometimes even a few hundred GB. WebA was developed by CompanyA and
provides basic searching capabilities, such as keyword search, faceting on
authors, classic stuff.

Now comes CompanyB with a project to build WebB, which should be a web
service built on existing WebA instances. The main goal is to take the
content of WebA, run some natural language processing on it, obtain
metadata about documents from external sources and index it, so the new
WebB will provide advanced searching capabilities on these added metadata.
WebB must also be able to search on fields from the original index, such as
author, title, but also be able to search on these new fields, effectively
combining the results.

Both WebA and WebB are for the same customer, so a bit of collaboration
between CompanyA and CompanyB is possible, however we should limit the
amount of work needed on WebA by CompanyA to minimum.

The issue is the following:
The first idea was that WebB will replicate the index of WebA and has its
own copy of the index. When new metadata is gathered for a document, it
will update the document in its own index. However, since both projects are
for the same customer, doubling the index was a bit of a problem. Also,
this idea was based on the presumption that the original index is not
updated very often, which we found out is not true.

The second approach was to use the existing index, so duplicating data on
two indexes would not be needed. But since the original index is used by
WebA and worked on by a different company, it would mean a lot of work for
them and it probably wouldn't be possible. On reindex, they would not have
access to new metadata that was posted into the index by CompanyB/WebB,
querying from WebB could affect performance on WebA, WebA would need to
filter the results to not return fields that were put there by WebB and so
on...

The third idea is probably the closest to reality, but it also might not
be. The idea was to create a new shard/node in the existing cluster of
WebA's Solr, where only the new metadata will be indexed. However, since we
need to be able to search and return results from both of these nodes
(queries like "author:ABC && metadata_one:123"), I believe it would also
not be possible. The documents in the new node containing only additional
metadata should be represented in the same document in a response, since we
want the result to contain both fields from original node (author, title,
...) and fields from new node (metadata1, namedEntities, ...). The results
of querying, faceting and sorting should also be somehow combined.

We also considered CDCR, effectively keeping a synchronized copy of the
original index but the use case for CDCR is a bit different, the target
cluster should not be updated without updates on the source cluster, which
is not what we want. We want to have an up-to-date original index enriched
with additional fields.


I hope my explanation is clear enough and I will appreciate your help.

Thank you for your time, and have a nice day :),
Norbert Bodnar

AW: Compacting JSON-Output

[Andreas Mock]

Hi Frank,

thank you for your fast answer.

Interesting. You referred to a parameter which is documented
for the XMLWriter, not for JSONWriter. But in the docs to 
version 6.6 one can see that it is used in the example
also with JSONWriter.

In the documents of the version we're trying to use 
(version 8.8; yes, I forgot to mention) there is also nothing said
about this parameter in the paragraph about JSONWriter.
But there it is also gone from the example config.

Just tried it and it seems to work.

So, next question: Why is this parameter documented with the
XMLWriter and not with the JSONWriter where it also seems 
to work?
Is there a kind of inheritance which is not clear to me?

Best regards
Andreas


> -Ursprüngliche Nachricht-
> Von: Nicolas Franck 
> Gesendet: Donnerstag, 15. April 2021 09:34
> An: users@solr.apache.org
> Betreff: Re: Compacting JSON-Output
> 
> https://solr.apache.org/guide/6_6/response-writers.html#ResponseWriters-
> TheindentParameter
> 
> > On 15 Apr 2021, at 09:32, Andreas Mock  wrote:
> >
> > Hi all,
> >
> > I searched around for a while now but couldn't find an answer to my
> problem.
> > We want to use the JSONWriter which presents a really nice human
> readable
> > output.
> >
> > Is there a way to force this writer to compact the JSON format as good
> as
> > possible to reduce the gross size of the reply?
> >
> > Best regards
> > Andreas Mock
> >

Re: Compacting JSON-Output

[Eric Pugh]

Andreas, I don’t think you are missing anything, the docs just need some 
updating….  

If you would like to create a JIRA issue and submit a pull request to update 
the docs for the JSONWriter, I’ll happily review and merge.  Please tag me 
(“David Eric Pugh”) on the ticket in JIRA.

Eric


> On Apr 15, 2021, at 4:25 AM, Andreas Mock  wrote:
> 
> Hi Frank,
> 
> thank you for your fast answer.
> 
> Interesting. You referred to a parameter which is documented
> for the XMLWriter, not for JSONWriter. But in the docs to 
> version 6.6 one can see that it is used in the example
> also with JSONWriter.
> 
> In the documents of the version we're trying to use 
> (version 8.8; yes, I forgot to mention) there is also nothing said
> about this parameter in the paragraph about JSONWriter.
> But there it is also gone from the example config.
> 
> Just tried it and it seems to work.
> 
> So, next question: Why is this parameter documented with the
> XMLWriter and not with the JSONWriter where it also seems 
> to work?
> Is there a kind of inheritance which is not clear to me?
> 
> Best regards
> Andreas
> 
> 
>> -Ursprüngliche Nachricht-
>> Von: Nicolas Franck 
>> Gesendet: Donnerstag, 15. April 2021 09:34
>> An: users@solr.apache.org
>> Betreff: Re: Compacting JSON-Output
>> 
>> https://solr.apache.org/guide/6_6/response-writers.html#ResponseWriters-
>> TheindentParameter
>> 
>>> On 15 Apr 2021, at 09:32, Andreas Mock  wrote:
>>> 
>>> Hi all,
>>> 
>>> I searched around for a while now but couldn't find an answer to my
>> problem.
>>> We want to use the JSONWriter which presents a really nice human
>> readable
>>> output.
>>> 
>>> Is there a way to force this writer to compact the JSON format as good
>> as
>>> possible to reduce the gross size of the reply?
>>> 
>>> Best regards
>>> Andreas Mock
>>> 
> 

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Re: Build Solr from Source - Failing Unit Tests in Intellij

[Jason Gerlowski]

Hi Phil,

Solr has a number of tests that are flaky and fail seemingly at
random.  Some of this is true flakiness: bugs that only occur with
certain timing behavior.  Some of it is driven by the Solr Test
Framework's heavy use of randomization in running test cases. ("ant
test" assigns seeds for test runs which are used to randomize aspects
of test execution: locale and timezone settings, the number of Solr
nodes or shards or documents used in certain tests, the client
settings used to talk to Solr, etc.)  Unfortunately, failures do
happen.

Ideally any failure would be investigated by developers as soon as it
occurs, but the extent of the randomization in play and the trickiness
of reproducing failures that may be timing-dependent means this often
isn't feasible in practice.

So where failures are reproducible, it's generally good practice to
let the community know what you found by reporting the failure on
"d...@solr.apache.org", or by recording the failure in a JIRA ticket
along with details about your environment and the "seed" used by the
test.  Or even better - by investigating yourself.

But where a failure isn't easily reproducible and you haven't made any
changes to related code, it's normal to let them go.

Best,

Jason

On Wed, Apr 14, 2021 at 3:20 PM Phill Campbell
 wrote:
>
> I have downloaded Solr 8.2.2 source code. I am using a Mac running Catalina.
> Using ANT with “clean compile test” everything is good.
>
> From Intellij I tried running tests and several failed.
> Here is an example:
>
> package org.apache.solr.analysis;
> ...
> public class TestWordDelimiterFilterFactory
> ...
>
> @Test
> public void testIgnoreCaseChange()
>
> ...
>
> assertQ("case change",
> req("subword:(good jon)")
> ,"//result[@numFound=1]"
> );
>
>
> Should I worry about that?
> What should I do?
>
> Thank you.

AW: Compacting JSON-Output

[Andreas Mock]

Hi Eric,

thank you for your comment.

As my learning curve for Solr itself is currently steep enough, I'm
not prepared to contribute in this way.

Best regards
Andreas


> -Ursprüngliche Nachricht-
> Von: Eric Pugh 
> Gesendet: Donnerstag, 15. April 2021 12:44
> An: users@solr.apache.org
> Betreff: Re: Compacting JSON-Output
> 
> Andreas, I don’t think you are missing anything, the docs just need some
> updating….
> 
> If you would like to create a JIRA issue and submit a pull request to
> update the docs for the JSONWriter, I’ll happily review and merge.  Please
> tag me (“David Eric Pugh”) on the ticket in JIRA.
> 
> Eric
> 
> 
> > On Apr 15, 2021, at 4:25 AM, Andreas Mock 
> wrote:
> >
> > Hi Frank,
> >
> > thank you for your fast answer.
> >
> > Interesting. You referred to a parameter which is documented
> > for the XMLWriter, not for JSONWriter. But in the docs to
> > version 6.6 one can see that it is used in the example
> > also with JSONWriter.
> >
> > In the documents of the version we're trying to use
> > (version 8.8; yes, I forgot to mention) there is also nothing said
> > about this parameter in the paragraph about JSONWriter.
> > But there it is also gone from the example config.
> >
> > Just tried it and it seems to work.
> >
> > So, next question: Why is this parameter documented with the
> > XMLWriter and not with the JSONWriter where it also seems
> > to work?
> > Is there a kind of inheritance which is not clear to me?
> >
> > Best regards
> > Andreas
> >
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Nicolas Franck 
> >> Gesendet: Donnerstag, 15. April 2021 09:34
> >> An: users@solr.apache.org
> >> Betreff: Re: Compacting JSON-Output
> >>
> >> https://solr.apache.org/guide/6_6/response-
> writers.html#ResponseWriters-
> >> TheindentParameter
> >>
> >>> On 15 Apr 2021, at 09:32, Andreas Mock 
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I searched around for a while now but couldn't find an answer to my
> >> problem.
> >>> We want to use the JSONWriter which presents a really nice human
> >> readable
> >>> output.
> >>>
> >>> Is there a way to force this writer to compact the JSON format as good
> >> as
> >>> possible to reduce the gross size of the reply?
> >>>
> >>> Best regards
> >>> Andreas Mock
> >>>
> >
> 
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com
>  | My Free/Busy
> 
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>  enterprise-search-server-third-edition-raw>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.

Building on Mac OS X Catalina

[Phill Campbell]

Is there a guide or recommendations for building on Mac OS Catalina?

Which JDK do you use?



This is one of the errors I currently get running the tests.

   [junit4]   2> 7123 WARN  
(SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
o.a.s.c.CoreContainer Unable to create 
[.../solr/core/src/test-files/solr/userfiles].  Features requiring this 
directory may fail.
   [junit4]   2>   => java.security.AccessControlException: access 
denied ("java.io.FilePermission” “.../solr/core/src/test-files/solr/userfiles" 
"write")
   [junit4]   2>at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)

suggester with a multivalued payload field

[Pratik Patel]

Hey Everyone,

I have a multivalued field which I am using as a payload field in a
suggester.

// field declaration

>  multiValued="true" />


// suggester component
> 
> 
>   insightSuggester
>   
>   
>name="lookupImpl">org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory
>   
>   DocumentDictionaryFactory
>   
>   concept_name
>   
>   is_named_entity_str
>   keyword_meta_type
>   is_named_entity_str
>   text_general
>   false
>   false
>   false
>   false
> 
> 


The payload field is a multi valued field and it has two values.

"keyword_meta_type":["DictionaryTerm", "NamedEntity"],
However when I query the suggester, I only get back one value with the
payload field.

{
"term":"maximal torus",
"weight":9,
"payload":"DictionaryTerm"

}


How can we make use of a multi valued field as a payload field in the suggester?


Thanks

Pratik

Re: Building on Mac OS X Catalina

[Shawn Heisey]

On 4/15/2021 10:38 AM, Phill Campbell wrote:

Is there a guide or recommendations for building on Mac OS Catalina?

Which JDK do you use?

This is one of the errors I currently get running the tests.

[junit4]   2> 7123 WARN  
(SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
o.a.s.c.CoreContainer Unable to create 
[.../solr/core/src/test-files/solr/userfiles].  Features requiring this directory 
may fail.
[junit4]   2>   => java.security.AccessControlException: access denied 
("java.io.FilePermission” “.../solr/core/src/test-files/solr/userfiles" "write")
[junit4]   2>at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)


The relevant part of that as far as I can see is that it is unable to 
create the following directory, where ... is probably the root of the 
cloned repository or extracted source tarball:


.../solr/core/src/test-files/solr/userfiles

Probably what's happening here is that you're running the tests with a 
different user than the one that owns the extracted or cloned files.  So 
when that particular unit test tries to create a directory, it fails. 
It's a basic file permission problem.


This would likely happen with ANY jdk, and it's not a Mac-specific problem.

Thanks,
Shawn

Re: Building on Mac OS X Catalina

[Phill Campbell]

Shawn,

Thank you for responding. I have “chmod’ed” the entire directory path for every 
directory.

[me ~/dev/mysolr/solr/core/src/test-files/solr]$ls -alh
total 160
drwxrwxrwx@ 28 me  COMPANY\Domain Users   896B Apr 15 10:06 .
drwxrwxrwx@ 15 me  COMPANY\Domain Users   480B Apr 15 09:37 ..
drwxr-xrwx@  3 me  COMPANY\Domain Users96B Apr  6 15:36 analysisconfs
drwxr-xrwx@  4 me  COMPANY\Domain Users   128B Apr  6 15:36 collection1
drwxr-xrwx@ 28 me  COMPANY\Domain Users   896B Apr  6 15:36 configsets
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.4K Apr  6 15:36 
crazy-path-to-config.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.5K Apr  6 15:36 
crazy-path-to-schema.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users71B Apr  6 15:36 external_eff
drwxr-xrwx@  6 me  COMPANY\Domain Users   192B Apr  6 15:36 
question-answer-repository
-rw-r--rw-@  1 me  COMPANY\Domain Users   497B Apr  6 15:36 
question-answer-repository-private-key.pem
drwxr-xrwx@ 10 me  COMPANY\Domain Users   320B Apr  6 15:36 security
drwxr-xrwx@  8 me  COMPANY\Domain Users   256B Apr  6 15:36 simSnapshot
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.5K Apr  6 15:36 solr-50-all.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.3K Apr  6 15:36 
solr-graphitereporter.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.2K Apr  6 15:36 
solr-hiddensysprops.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.1K Apr  6 15:36 solr-jmxreporter.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.8K Apr  6 15:36 
solr-metricreporter.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.3K Apr  6 15:36 
solr-metricsconfig.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.1K Apr  6 15:36 
solr-metricsconfig1.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.1K Apr  6 15:36 
solr-shardhandler-loadBalancerRequests.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.1K Apr  6 15:36 
solr-shardhandler.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.8K Apr  6 15:36 
solr-slf4jreporter.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   947B Apr  6 15:36 
solr-solrDataHome.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   3.3K Apr  6 15:36 
solr-solrreporter.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   1.3K Apr  6 15:36 solr-stress-new.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.5K Apr  6 15:36 solr-tracing.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.1K Apr  6 15:36 
solr-trackingshardhandler.xml
-rw-r--rw-@  1 me  COMPANY\Domain Users   2.8K Apr  6 15:36 solr.xml

Another thing I have found. The test that fails with the creation of the 
userfiles using ANT works from Intellij IDEA.

Regards.

> On Apr 15, 2021, at 11:00 AM, Shawn Heisey  wrote:
> 
> On 4/15/2021 10:38 AM, Phill Campbell wrote:
>> Is there a guide or recommendations for building on Mac OS Catalina?
>> Which JDK do you use?
>> This is one of the errors I currently get running the tests.
>>[junit4]   2> 7123 WARN  
>> (SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
>> o.a.s.c.CoreContainer Unable to create 
>> [.../solr/core/src/test-files/solr/userfiles].  Features requiring this 
>> directory may fail.
>>[junit4]   2>   => java.security.AccessControlException: access 
>> denied ("java.io.FilePermission” 
>> “.../solr/core/src/test-files/solr/userfiles" "write")
>>[junit4]   2> at 
>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> 
> The relevant part of that as far as I can see is that it is unable to create 
> the following directory, where ... is probably the root of the cloned 
> repository or extracted source tarball:
> 
> .../solr/core/src/test-files/solr/userfiles
> 
> Probably what's happening here is that you're running the tests with a 
> different user than the one that owns the extracted or cloned files.  So when 
> that particular unit test tries to create a directory, it fails. It's a basic 
> file permission problem.
> 
> This would likely happen with ANY jdk, and it's not a Mac-specific problem.
> 
> Thanks,
> Shawn

Re: Building on Mac OS X Catalina

[Mike Drob]

OpenJDK 11 is fine for building the main (9x) branch.

This is probably https://issues.apache.org/jira/browse/SOLR-14084 or
https://issues.apache.org/jira/browse/SOLR-15222 and happens sometimes
when the test randomization includes the security policy.

It's something I've been wanting to look at but keeps getting pushed
for other priorities.

On Thu, Apr 15, 2021 at 11:39 AM Phill Campbell
 wrote:
>
> Is there a guide or recommendations for building on Mac OS Catalina?
>
> Which JDK do you use?
>
>
>
> This is one of the errors I currently get running the tests.
>
>[junit4]   2> 7123 WARN  
> (SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
> o.a.s.c.CoreContainer Unable to create 
> [.../solr/core/src/test-files/solr/userfiles].  Features requiring this 
> directory may fail.
>[junit4]   2>   => java.security.AccessControlException: access 
> denied ("java.io.FilePermission” 
> “.../solr/core/src/test-files/solr/userfiles" "write")
>[junit4]   2>at 
> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)

Re: Building on Mac OS X Catalina

[Phill Campbell]

Mike,
I think you are on to it.
I am building Solr 8.2.1 using OpenJDK 

openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-bre_2021_01_20_16_37-b00)


How is a security policy used in the ANT build? I am unfamiliar with ANT. I 
will gladly change something to see if it works. (I have been using Maven and 
Gradle).

I am willing to try any suggestions.

Thanks you.




> On Apr 15, 2021, at 3:44 PM, Mike Drob  wrote:
> 
> OpenJDK 11 is fine for building the main (9x) branch.
> 
> This is probably https://issues.apache.org/jira/browse/SOLR-14084 or
> https://issues.apache.org/jira/browse/SOLR-15222 and happens sometimes
> when the test randomization includes the security policy.
> 
> It's something I've been wanting to look at but keeps getting pushed
> for other priorities.
> 
> On Thu, Apr 15, 2021 at 11:39 AM Phill Campbell
>  wrote:
>> 
>> Is there a guide or recommendations for building on Mac OS Catalina?
>> 
>> Which JDK do you use?
>> 
>> 
>> 
>> This is one of the errors I currently get running the tests.
>> 
>>   [junit4]   2> 7123 WARN  
>> (SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
>> o.a.s.c.CoreContainer Unable to create 
>> [.../solr/core/src/test-files/solr/userfiles].  Features requiring this 
>> directory may fail.
>>   [junit4]   2>   => java.security.AccessControlException: access 
>> denied ("java.io.FilePermission” 
>> “.../solr/core/src/test-files/solr/userfiles" "write")
>>   [junit4]   2>at 
>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)

Re: Building on Mac OS X Catalina

[Phill Campbell]

Sorry, meant to say 8.8.2 Solr.

I read those JIRA reports, and found in common-build.xml:




I set that to be false and the exception is gone from the output of the tests.



> On Apr 15, 2021, at 3:44 PM, Mike Drob  wrote:
> 
> OpenJDK 11 is fine for building the main (9x) branch.
> 
> This is probably https://issues.apache.org/jira/browse/SOLR-14084 or
> https://issues.apache.org/jira/browse/SOLR-15222 and happens sometimes
> when the test randomization includes the security policy.
> 
> It's something I've been wanting to look at but keeps getting pushed
> for other priorities.
> 
> On Thu, Apr 15, 2021 at 11:39 AM Phill Campbell
>  wrote:
>> 
>> Is there a guide or recommendations for building on Mac OS Catalina?
>> 
>> Which JDK do you use?
>> 
>> 
>> 
>> This is one of the errors I currently get running the tests.
>> 
>>   [junit4]   2> 7123 WARN  
>> (SUITE-TestCircuitBreaker-seed#[597FF1CD7ECC0E37]-worker) [ ] 
>> o.a.s.c.CoreContainer Unable to create 
>> [.../solr/core/src/test-files/solr/userfiles].  Features requiring this 
>> directory may fail.
>>   [junit4]   2>   => java.security.AccessControlException: access 
>> denied ("java.io.FilePermission” 
>> “.../solr/core/src/test-files/solr/userfiles" "write")
>>   [junit4]   2>at 
>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)