Re: Security: Better secure defaults?
I would like to be able to define core specific permissions with rule-based authorization in security.json in the same way you can do for collections. Thomas Op do 6 mei 2021 om 23:25 schreef David Smiley : > I'm reaching out to our user community to get opinions on what Solr should > do to be more secure-by-default. > > TL;DR: Solr 9 has better secure-by-defaults, but maybe we should do more > like have Solr pick some of it's default settings dependent on a new > env=dev|prod. > > I was shown a glimpse of a massive list of Solr servers exposed on the > public internet by a security researcher. I'm kinda blown away that so > many people would be so careless. I think Solr could and should run with > better "secure-by-default" settings. > > The situation will be much better in Solr 9 -- and I'll give a shout-out of > thanks to Rob Muir for helping make this so. Here's a couple prominent > ones: > * Solr's Jetty now binds to localhost by default, configurable via > SOLR_JETTY_HOST. Before 9, you can configure a similar thing in the Jetty > config files. SOLR-13985 > * Java's SecurityManager sandbox is enabled by default. -- SOLR-13984. > This option also exists in Solr since 8.5, toggle-able > via SOLR_SECURITY_MANAGER_ENABLED. Mostly this prevents the worst of > security bugs -- RCE. > > I wonder if users will promptly set SOLR_JETTY_HOST=0.0.0.0 to get anything > done? I think so... but it's something, protecting some users. > > Perhaps Solr ought to default to requiring a username/password? I've heard > this suggestion and it's an obvious one even if some of us (me included) > worry that it would make it too annoying to play with Solr when getting > started. I think the concerns could be mitigated based on the approach. > If Solr had an opt-in env=dev setting, for example, then Solr could not > insist on authentication, whereas a default env=prod would insist. Of > course the authentication or lack thereof could be explicitly configured or > disabled at the user's prerogative. What I like about an "env" setting is > that many other settings could be gated on this as well. > > I particularly like the idea of an env=dev|prod setting because a variety > of settings in Solr could have a default that is dependent on this value. > In particular I argue that a env=prod should result in Solr's config APIs > being disabled -- equivalent to -Ddisable.configEdit=true. I believe a > minority of Solr users actually use these APIs, yet they are frequently a > step in exploiting weaknesses in Solr. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley >
Re: Content search and applying ACL
After upgrading our Solr to 8.8 from 7.2, I've tested out the *score=none*, and *method=toplevelDV* with two collections. The first few queries took even longer, the first one took up to 20s as compared to 8s from *method=index*. I guess this is inline with what the docs mentioned about topLevelDV, "These data structures outperform other methods as the number of values matched in the from field grows high. But they are also expensive to build and need to be lazily populated after each commit, causing a sometimes-noticeable slowdown on the first query to use them after each commit." What I'm now curious is what these "data structures that are expensive to build" are? I dived into /TopLevelJoinQuery.java/, and saw that the implementation retrieves the DocValues from the respective indexes and use them for the join operation. Am I right to assume that the "data structures that are expensive to build" is referring to DocValues? Would that also mean that other operations (e.g. faceting) that use DocValues will take some time after a commit? Anyway, to solve the problem detailed in the first post, we denormalised our indexes (i.e. embed the list of teams that can read the information into our content documents), and lookups are now almost instantaneous. One consequence of this is that whenever the ACL changes (i.e. which teams can read this content), we'll have to re-index all relevant content data. We have judged this to be acceptable, because 1) We can run this asynchronously 2) ACL changes need not be instantaneous 3) Due to the nature of our app, the amount time required to re-index the relevant content data is still within acceptable range and will be difficult to grow out of unacceptable range. Hope this helps any individual who comes across this! :) -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr equivalent of relational joins on nested documents
Thanks! I am aware of the examples in the docs but they don't quite answer my
specific question about parent/child joins on multiple levels in a general way.
Here is a slightly modified example that actually works on ElasticSearch with
an equivalent (nested) schema. I can't figure out how to express that in Solr,
either with the Lucene QParser or the JSON request syntax. Only the streaming
requests seem to make it possible but it's an entirely different syntax and
brings its own issues, I'd rather not get into that just for expressing
filtering conditions on an extra level.
Cheers
{
"query": {
"nested": {
"path": "level1",
"query": {
"bool": {
"must": [
{
"match": {
"level1.field1": "jcqptr1"
}
},
{
"nested": {
"path": "level1.level2",
"query": {
"bool": {
"must": [
{
"match": {
"level2.field1": "SVE"
}
},
{
"match": {
"level2.field2": "GPP"
}
},
{
"match": {
"level2.field3": "PRE"
}
},
{
"match": {
"level2.field4": "mqnmnq"
}
}
]
}
}
}
}
]
}
}
}
}
}
On 2021/05/05 09:05:37, Norbert Kutasi wrote:
> Hello Alain,
>
> If I was going to implement queries on deeply nested documents, I can see
> quite a few examples here :
> https://solr.apache.org/guide/8_8/searching-nested-documents.html#parent-query-parser
>
>
> In a hierarchy like below the following query with a criteria on
> _nest_path_ suppose to retrieve parent documents (Products) based on
> attributes of Manual: * q={!parent which="*:*
> -_nest_path_:*"}(+_nest_path_:\/skus\/manuals +pages_i:1)*
> Product
> SKU
> manuals
>
> In the "fl" parameter then you would probably need to initiate *[child] -
> ChildDocTransformerFactory* to retrieve the hierarchy with essentially
> repeating the query chain to avoid revealing child documents that are not
> matching with parameters sent in "q".
>
> Sorry if that comes off as clumsy or trivial, but in our implementation, we
> do not index documents in a Parent / Child way. In order to generate nested
> JSON responses, we use [subquery] document transformer , that does not go
> with BlockJoin Parsers, so when it comes to filtering we rely on some
> denormalized set of attributes available for all objects (not in RDBMS
> source system but in SOLR), alternatively at query time the hierarchy would
> change say Steps -> Interactions -> Customers (to use your analogy) if
> filtering needed for (a leaf) object specific attribute.
>
> Norbert
>
> On Tue, 4 May 2021 at 21:01, Timothy Potter wrote:
>
> > Did you try routing all docs for the same customer ID to the same
> > shard (not block join), just docs of different types in the same shard
> > based on customer ID and then using Joins?
> >
> > On Tue, May 4, 2021 at 4:06 AM Alain Rogister
> > wrote:
> > >
> > > It sounds like nobody knows. Just one last try. Up !
> > >
> > > Tx
> > >
> > > On 2021/04/21 10:08:16, Alain Rogister wrote:
> > > > Hi,
> > > >
> > > > Imagine that you have been tasked with designing a Solr-based system
> > that indexes and queries data aggregated from several relational databases.
> > All the data items are ultimately related to a common object type, so the
> > logical data schema looks like :
> > > >
> > > > Customer
> > > > Profile
> > > > Events
> > > > Interactions
> > > >Steps
> > > > ActionPlans
> > > >Objectives
> > > > Actions
> > > > (and quite a few more, with up to 3 or 4 levels of nesting)
> > > >
> >
Solr: Get leaderness status of local node
Hi Solr people,
I am writing a custom UpdateProcessor, part of a custom plugin, and need to run
some code only on the shard leader. This is a plugin, so I cannot access the
DistributedUpdateProcessor.isLeader() method which is not public.
For now I am copying-pasting the below code, but I am thinking there's got to
be a better way to do it.
private boolean getIsLeader() {
final boolean isZkAware =
req.getCore().getCoreContainer().isZooKeeperAware();
if (!isZkAware) {
return getNonZkLeaderAssumption(req);
}
String shardId = cloudDesc.getShardId();
try {
Replica leaderReplica =
zkController.getZkStateReader().getLeaderRetry(collection, shardId);
return leaderReplica.getName().equals(cloudDesc.getCoreNodeName());
}
catch (InterruptedException e) {
throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR,
“Error TODO", e);
}
}
I think there is also another way to do it using cloudDesc.isLeader() but my
understanding, if I am not wrong, is that the first code gives the most
accurate state. Am I right? Is this the only way to get the most accurate state
about the current leader?
Also, should I run it every time I need to check the current status as the
leader can change anytime? What's the impact in terms of performance?
Thank you for your help in advance.
Lamine
Re: Security: Better secure defaults?
I didn't propose a new configuration file, I proposed a "mode". Even if there was some new env-specific file, it would end up being yet another configuration file -- no thanks. Solr still needs solr.xml & solrconfig.xml & schema.xml (and more), plus Jetty's many config files, and they are mostly for different purposes. Even if we were to hypothesize a brand new system that had one configuration file, what does the code reading this file do if the particular named setting isn't present? *That* question is where a "mode" might in some cases yield a different default depending on the setting. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Thu, May 6, 2021 at 7:02 PM Eric Pugh wrote: > I think that part of the challenge goes to the deeper issue that > configuring Solr isn’t easy. We don’t really have the concept of a > environment specific settings file. I’d love to see a -env=production.yml > or -env=development.yml type file that was the single place for all > settings, and had sane defaults for each environment. Something that > worked across Docker, classic installed service, or just via a bin/solr > start command line ;-). > > I am constantly finding new command line config options that I didn’t know > about ;-) > > > > On May 6, 2021, at 5:58 PM, matthew sporleder > wrote: > > > > On Thu, May 6, 2021 at 5:25 PM David Smiley wrote: > >> > >> I'm reaching out to our user community to get opinions on what Solr > should > >> do to be more secure-by-default. > >> > >> TL;DR: Solr 9 has better secure-by-defaults, but maybe we should do more > >> like have Solr pick some of it's default settings dependent on a new > >> env=dev|prod. > >> > >> I was shown a glimpse of a massive list of Solr servers exposed on the > >> public internet by a security researcher. I'm kinda blown away that so > >> many people would be so careless. I think Solr could and should run > with > >> better "secure-by-default" settings. > >> > >> The situation will be much better in Solr 9 -- and I'll give a > shout-out of > >> thanks to Rob Muir for helping make this so. Here's a couple prominent > >> ones: > >> * Solr's Jetty now binds to localhost by default, configurable via > >> SOLR_JETTY_HOST. Before 9, you can configure a similar thing in the > Jetty > >> config files. SOLR-13985 > >> * Java's SecurityManager sandbox is enabled by default. -- SOLR-13984. > >> This option also exists in Solr since 8.5, toggle-able > >> via SOLR_SECURITY_MANAGER_ENABLED. Mostly this prevents the worst of > >> security bugs -- RCE. > >> > >> I wonder if users will promptly set SOLR_JETTY_HOST=0.0.0.0 to get > anything > >> done? I think so... but it's something, protecting some users. > >> > >> Perhaps Solr ought to default to requiring a username/password? I've > heard > >> this suggestion and it's an obvious one even if some of us (me included) > >> worry that it would make it too annoying to play with Solr when getting > >> started. I think the concerns could be mitigated based on the approach. > >> If Solr had an opt-in env=dev setting, for example, then Solr could not > >> insist on authentication, whereas a default env=prod would insist. Of > >> course the authentication or lack thereof could be explicitly > configured or > >> disabled at the user's prerogative. What I like about an "env" setting > is > >> that many other settings could be gated on this as well. > >> > >> I particularly like the idea of an env=dev|prod setting because a > variety > >> of settings in Solr could have a default that is dependent on this > value. > >> In particular I argue that a env=prod should result in Solr's config > APIs > >> being disabled -- equivalent to -Ddisable.configEdit=true. I believe a > >> minority of Solr users actually use these APIs, yet they are frequently > a > >> step in exploiting weaknesses in Solr. > >> > >> ~ David Smiley > >> Apache Lucene/Solr Search Developer > >> http://www.linkedin.com/in/davidwsmiley > > > > I have also found open solr servers and normally send an email like > > "shall I delete your data or wait for you to do it?" > > > > As solr is primarily an index of other data the primary issue is data > > disclosure. Config editing, inserting data, etc are all pretty > > secondary. > > HTTP Basic Auth with a first-boot-random password is a massively > > simple thing built into jetty that will solve 99% of exposed solr > > servers. > > > > secure-by-default will decrease adoption without major east-to-follow > > warning messages so tread lightly. > > ___ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com < > http://www.opensourceconnections.com/> | My Free/Busy < > http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed < > https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> > > This e-
Re: Security: Better secure defaults?
> Listen on 0.0.0.0 but only accept traffic from private addresses? Respect x-forwarded-for (and its aliases) in that case. +1 ! Feel free to file a JIRA issue; maybe there is one already. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Thu, May 6, 2021 at 8:39 PM matthew sporleder wrote: > One problem I have seen in the past is cultural. Back when solr was > mostly a .war file it was very easy to say "secure your own tomcat" but in > the era of "solr is a database not a web app" it needs to embrace the > entirety of that distinction. > > You can't have it both ways and I am not sure the culture shift was fully > embraced. > > Another point of feedback is that we have built apps with both solr > "clients" and a lot of diy with curl so any security settings need to be > sane. > > Listen on 0.0.0.0 but only accept traffic from private addresses? Respect > x-forwarded-for (and its aliases) in that case. > > > > > > > On May 6, 2021, at 7:02 PM, Eric Pugh > wrote: > > > > I think that part of the challenge goes to the deeper issue that > configuring Solr isn’t easy. We don’t really have the concept of a > environment specific settings file. I’d love to see a -env=production.yml > or -env=development.yml type file that was the single place for all > settings, and had sane defaults for each environment. Something that > worked across Docker, classic installed service, or just via a bin/solr > start command line ;-). > > > > I am constantly finding new command line config options that I didn’t > know about ;-) > > > > > >> On May 6, 2021, at 5:58 PM, matthew sporleder > wrote: > >> > >>> On Thu, May 6, 2021 at 5:25 PM David Smiley > wrote: > >>> > >>> I'm reaching out to our user community to get opinions on what Solr > should > >>> do to be more secure-by-default. > >>> > >>> TL;DR: Solr 9 has better secure-by-defaults, but maybe we should do > more > >>> like have Solr pick some of it's default settings dependent on a new > >>> env=dev|prod. > >>> > >>> I was shown a glimpse of a massive list of Solr servers exposed on the > >>> public internet by a security researcher. I'm kinda blown away that so > >>> many people would be so careless. I think Solr could and should run > with > >>> better "secure-by-default" settings. > >>> > >>> The situation will be much better in Solr 9 -- and I'll give a > shout-out of > >>> thanks to Rob Muir for helping make this so. Here's a couple prominent > >>> ones: > >>> * Solr's Jetty now binds to localhost by default, configurable via > >>> SOLR_JETTY_HOST. Before 9, you can configure a similar thing in the > Jetty > >>> config files. SOLR-13985 > >>> * Java's SecurityManager sandbox is enabled by default. -- SOLR-13984. > >>> This option also exists in Solr since 8.5, toggle-able > >>> via SOLR_SECURITY_MANAGER_ENABLED. Mostly this prevents the worst of > >>> security bugs -- RCE. > >>> > >>> I wonder if users will promptly set SOLR_JETTY_HOST=0.0.0.0 to get > anything > >>> done? I think so... but it's something, protecting some users. > >>> > >>> Perhaps Solr ought to default to requiring a username/password? I've > heard > >>> this suggestion and it's an obvious one even if some of us (me > included) > >>> worry that it would make it too annoying to play with Solr when getting > >>> started. I think the concerns could be mitigated based on the > approach. > >>> If Solr had an opt-in env=dev setting, for example, then Solr could not > >>> insist on authentication, whereas a default env=prod would insist. Of > >>> course the authentication or lack thereof could be explicitly > configured or > >>> disabled at the user's prerogative. What I like about an "env" > setting is > >>> that many other settings could be gated on this as well. > >>> > >>> I particularly like the idea of an env=dev|prod setting because a > variety > >>> of settings in Solr could have a default that is dependent on this > value. > >>> In particular I argue that a env=prod should result in Solr's config > APIs > >>> being disabled -- equivalent to -Ddisable.configEdit=true. I believe a > >>> minority of Solr users actually use these APIs, yet they are > frequently a > >>> step in exploiting weaknesses in Solr. > >>> > >>> ~ David Smiley > >>> Apache Lucene/Solr Search Developer > >>> http://www.linkedin.com/in/davidwsmiley > >> > >> I have also found open solr servers and normally send an email like > >> "shall I delete your data or wait for you to do it?" > >> > >> As solr is primarily an index of other data the primary issue is data > >> disclosure. Config editing, inserting data, etc are all pretty > >> secondary. > >> HTTP Basic Auth with a first-boot-random password is a massively > >> simple thing built into jetty that will solve 99% of exposed solr > >> servers. > >> > >> secure-by-default will decrease adoption without major east-to-follow > >> warning messages so tread lightly. > > > > ___ > > Eric Pugh |
Re: Security: Better secure defaults?
> I would like to be able to define core specific permissions with rule-based > authorization in security.json in the same way you can do for collections. PRs/Patches welcome... but I think you're going to have to accept migrating to SolrCloud. SolrCloud has gotten better year over year. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Fri, May 7, 2021 at 3:39 AM Thomas Corthals wrote: > I would like to be able to define core specific permissions with rule-based > authorization in security.json in the same way you can do for collections. > > Thomas > > Op do 6 mei 2021 om 23:25 schreef David Smiley : > > > I'm reaching out to our user community to get opinions on what Solr > should > > do to be more secure-by-default. > > > > TL;DR: Solr 9 has better secure-by-defaults, but maybe we should do more > > like have Solr pick some of it's default settings dependent on a new > > env=dev|prod. > > > > I was shown a glimpse of a massive list of Solr servers exposed on the > > public internet by a security researcher. I'm kinda blown away that so > > many people would be so careless. I think Solr could and should run with > > better "secure-by-default" settings. > > > > The situation will be much better in Solr 9 -- and I'll give a shout-out > of > > thanks to Rob Muir for helping make this so. Here's a couple prominent > > ones: > > * Solr's Jetty now binds to localhost by default, configurable via > > SOLR_JETTY_HOST. Before 9, you can configure a similar thing in the > Jetty > > config files. SOLR-13985 > > * Java's SecurityManager sandbox is enabled by default. -- SOLR-13984. > > This option also exists in Solr since 8.5, toggle-able > > via SOLR_SECURITY_MANAGER_ENABLED. Mostly this prevents the worst of > > security bugs -- RCE. > > > > I wonder if users will promptly set SOLR_JETTY_HOST=0.0.0.0 to get > anything > > done? I think so... but it's something, protecting some users. > > > > Perhaps Solr ought to default to requiring a username/password? I've > heard > > this suggestion and it's an obvious one even if some of us (me included) > > worry that it would make it too annoying to play with Solr when getting > > started. I think the concerns could be mitigated based on the approach. > > If Solr had an opt-in env=dev setting, for example, then Solr could not > > insist on authentication, whereas a default env=prod would insist. Of > > course the authentication or lack thereof could be explicitly configured > or > > disabled at the user's prerogative. What I like about an "env" setting > is > > that many other settings could be gated on this as well. > > > > I particularly like the idea of an env=dev|prod setting because a variety > > of settings in Solr could have a default that is dependent on this value. > > In particular I argue that a env=prod should result in Solr's config APIs > > being disabled -- equivalent to -Ddisable.configEdit=true. I believe a > > minority of Solr users actually use these APIs, yet they are frequently a > > step in exploiting weaknesses in Solr. > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > http://www.linkedin.com/in/davidwsmiley > > >
Re: Solr 8.8.2 childFilter multiple conditions
Subquery is a good idea here too. It's slower but it's also more powerful,
more general.
I'll try and get SOLR-15156 into 8.9
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, May 6, 2021 at 3:42 PM Jonathan Bridges
wrote:
> Thanks David for confirming this and the work around, really appreciate it.
>
> I also found a subquery could give correct results, though haven't tested
> performance comparison with childFilter. I had never used subquery before
> so it hadn't occurred to me. Will try them both out.
>
> fq={!parent which='docType:factory'} docType:widget AND color:Red AND
> size:Large
> fl=*, widgets:[subquery]
> widgets.q=docType: child_doc_type AND color: Red AND size: Large
> widgets.fq={!terms f=_nest_parent_ v=$row.id}
>
> Thanks again,
> Jon Bridges
>
>
> On Thu, May 6, 2021 at 9:38 AM David Smiley wrote:
>
> > Hi,
> >
> > It appears this was broken in 8.0 by
> > https://issues.apache.org/jira/browse/SOLR-12768
> > And then fixed in 9.0 (not released) by SOLR-15156
> >
> > I didn't back-port the fix because I thought users might be relying on
> the
> > escaping behavior. I didn't realize such escaping didn't exist before
> > SOLR-12768. In light of this, the change would make an appropriate
> > addition to 8.9 but not a bug-fix version.
> >
> > A work-around is to express your query without colons, since the internal
> > logic is gated on that. So something like:
> >childFilter='+{!field f=docType v=child_doc_type} +{!field f=color
> > v=Red}'etc. Disclaimer: I didn't test that; it *may* be
> necessary
> > to move out some of this into other parameters based on parsing
> > rules/restrictions e.g. childFilter=$theChildFilter and then add a
> > top-level param. But I think it's not needed.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Tue, May 4, 2021 at 6:30 PM Jonathan Bridges
> > wrote:
> >
> > > Hello,
> > >
> > > With Solr 8.8.2 I am unable to provide more than one filter in the fl
> > > "childFilter" param.
> > >
> > > For example,
> > >
> > > fq={!parent which='docType:parent_doc_type'} docType: child_doc_type
> AND
> > > color: Red AND size: Large
> > >
> > > This fq will filter all parent documents which have child documents
> that
> > > are 'Red' and 'Large'.
> > >
> > > Now I want to show the parents with the children in the result, but
> only
> > > the children which are 'Red' and 'Large'. In the past (Solr v6) I would
> > > accomplish this with:
> > >
> > > fl=*, [child childFilter='docType:child_doc_type AND color: Red AND
> size:
> > > Large']
> > > However, with Solr 8.8.2 I get no children in the results and no
> errors.
> > >
> > > If I remove the extra AND conditions:
> > >
> > > fl=*, [child childFilter='docType:child_doc_type']
> > > I then get all child documents, even ones which are Blue and Small.
> > >
> > > How can I include only child documents which match the fq? Is there
> some
> > > way to specify multiple conditions now that I just can't find in the
> > docs?
> > >
> > > Thanks for any help.
> > >
> >
>
