[
https://issues.apache.org/jira/browse/METRON-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444396#comment-16444396
]
ASF GitHub Bot commented on METRON-1526:
----------------------------------------
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/995
It might also be possible to do this the other way, rather than getting a
list of all fields then figuring out which are polyFields, we might be able to
get just all valid fields. Substantially more validation would have to be done
on what I've tested here though to make sure it's valid and I didn't skip over
something important.
I threw this together in one of the metaalerts tests:
```
MetronSolrClient client = solr.getSolrClient();
client.setDefaultCollection("test");
LukeRequest lukeRequest = new LukeRequest();
lukeRequest.setShowSchema(true);
LukeResponse result = lukeRequest
.process(client);
Map<String, FieldInfo> fieldInfo = result.getFieldInfo();
for(Entry<String, FieldInfo> entry : fieldInfo.entrySet()) {
System.out.println("KEY: " + entry.getKey());
System.out.println("VALUE NAME: " + entry.getValue().getName());
System.out.println("FLAGS: " + entry.getValue().getFlags());
System.out.println();
}
```
Output is
>KEY: new-field
>VALUE NAME: new-field
>FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
>
>KEY: score
>VALUE NAME: score
>FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]
>
>KEY: _version_
>VALUE NAME: _version_
>FLAGS: [DOC_VALUES]
>
>KEY: threat:triage:score
>VALUE NAME: threat:triage:score
>FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]
>
>KEY: _root_
>VALUE NAME: _root_
>FLAGS: [INDEXED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
>
>KEY: location_point
>VALUE NAME: location_point
>FLAGS: [INDEXED, STORED, OMIT_TF]
>
>KEY: name
>VALUE NAME: name
>FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
>
>KEY: guid
>VALUE NAME: guid
>FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
>
>KEY: metaalerts
>VALUE NAME: metaalerts
>FLAGS: [INDEXED, STORED, MULTI_VALUED, OMIT_NORMS, OMIT_TF,
SORT_MISSING_LAST]
>
>KEY: source.type
>VALUE NAME: source.type
>FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
>
>KEY: timestamp
>VALUE NAME: timestamp
>FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]
Notice there's no "_coordinate" fields. Having "location_point" defined in
my schema with:
```
<field name="location_point" type="location" indexed="true"
stored="true"/>
<dynamicField name="*_coordinate" type="pdouble" indexed="true"
stored="false"/>
...
<fieldType name="location" class="solr.LatLonType"
subFieldSuffix="_coordinate"/>
```
Resulted in the "_coordinate" fields being present in this query:
```
client = solr.getSolrClient();
client.setDefaultCollection("test");
SolrQuery query = new SolrQuery("*:*");
query.set("fl", "*");
QueryResponse response = client.query(query);
System.out.println(response.getResults());
```
Like so (with all the extra stripped out so it's easier to read here):
```
{
...,
docs=[
SolrDocument{threat:triage:score=0.0, location_point=10,20,
guid=message_0, metaalerts=[meta_active], source.type=test,
timestamp=1524157278555, ip_src_port=8010, location_point_1_coordinate=20.0,
_version_=1598194742551642112, location_point_0_coordinate=10.0,
ip_src_addr=192.168.1.1},
...
]}
```
There is a catch: the schema has to be updated and I'm not sure it's
compatible with both 5.x and 6.x
`<requestHandler name="/admin/luke"
class="org.apache.solr.handler.admin.LukeRequestHandler" />`
> Location field types cause DocValuesField appear more than once error
> ---------------------------------------------------------------------
>
> Key: METRON-1526
> URL: https://issues.apache.org/jira/browse/METRON-1526
> Project: Metron
> Issue Type: Bug
> Reporter: Ryan Merriman
> Assignee: Ryan Merriman
> Priority: Major
>
> While testing [https://github.com/apache/metron/pull/970] I get this error
> when creating a meta alert:
> {code:java}
> Error from server at http://10.0.2.15:8983/solr/bro: Exception writing
> document id bbc150f5-92f8-485d-93cc-11730c1edf31 to the index; possible
> analysis error: DocValuesField
> \"enrichments.geo.ip_dst_addr.location_point_0_coordinate\" appears more than
> once in this document (only one value is allowed per field){code}
> I tracked it down to the fact that multiple fields are returned for a
> location field. For example when a field named
> "enrichments.geo.ip_dst_addr.location_point" is configured in a schema, these
> fields are returned in a query:
> {code:java}
> {
> "enrichments.geo.ip_dst_addr.location_point_0_coordinate": "33.4499",
> "enrichments.geo.ip_dst_addr.location_point_1_coordinate": "-112.0712",
> "enrichments.geo.ip_dst_addr.location_point": "33.4499,-112.0712"
> }
> {code}
> We need a way to either suppress these extra fields when querying or remove
> them before updating a document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
