[
https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517143#comment-17517143
]
Otto Fowler edited comment on NIFI-9863 at 4/4/22 10:45 PM:
------------------------------------------------------------
[I'm going to keep this Grok specific, but there is a generic class of
operation here I *think*]
On the controller and its configuration side:
* Allow entering in of grok patterns with test and validation
* Allow entering of grok expressions with test and validation
* Allow grouping of expressions into logical sets
* Allow schema discovery from expressions and association with the expression
* the schema of a set needs to be the same for each expression in the set
* possibly schema for a pattern as well
So, let me manage my grok patterns in a reusable way, and help me to make sure
that they do what I think they do, and that the schema is know so I don't mess
that up.
from the consumer?
List<PatternObjectMayHaveSchema> getPatterns()
List<String> getAvailablePatterns()
PatternObjectMayHaveSchema getPattern(String name)
List<ExpressionObjectMayHaveSchema> getExpressions()
List<String> getAvailableExpressions()
ExpressionObjectMayHaveSchema getExpression(String name)
List<ExpressionSet> getExpressionSets()
List<String> getAvailableExpressionSets()
ExpressionSet getExpressionSet(String name)
We can also have a cache for compiled expressions ( that times out ) associated
was (Author: ottobackwards):
[I'm going to keep this Grok specific, but there is a generic class of
operation here I *think*]
On the controller and it's configuration side:
* Allow entering in of grok patterns with test and validation
* Allow entering of grok expressions with test and validation
* Allow grouping of expressions into logical sets
* Allow schema discovery from expressions and association with the expression
* the schema of a set needs to be the same for each expression in the set
* possibly schema for a pattern as well
So, let me manage my grok patterns in a reusable way, and help me to make sure
that they do what I think they do, and that the schema is know so I don't mess
that up.
from the consumer?
List<PatternObjectMayHaveSchema> getPatterns()
List<String> getAvailablePatterns()
PatternObjectMayHaveSchema getPattern(String name)
List<ExpressionObjectMayHaveSchema> getExpressions()
List<String> getAvailableExpressions()
ExpressionObjectMayHaveSchema getExpression(String name)
List<ExpressionSet> getExpressionSets()
List<String> getAvailableExpressionSets()
ExpressionSet getExpressionSet(String name)
We can also have a cache for compiled expressions ( that times out ) associated
> Controller Service for managing custom Grok patterns
> ----------------------------------------------------
>
> Key: NIFI-9863
> URL: https://issues.apache.org/jira/browse/NIFI-9863
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Otto Fowler
> Priority: Major
>
> Managing custom Grok expressions in properties for the Grok processors or
> Record readers is cumbersome and not ideal.
> Having a service that managed these expressions in a centralized and reusable
> way would be a benefit to those using Grok patterns.
> This service would allow the configuration of some number custom Grok
> patterns as the service configuration. The MVP would be manual entry, but
> loading patterns from File ( upload to configuration? ) or from some external
> location could be allowed as well down the line.
> In use, it could be argued that the patterns should be loaded from something
> like the schema registry.
> consumers of the service should then be able select the specific service
> instance and then using dependent properties select which patterns provided
> by the service to consume.
> To this end, it may be nice to have the service support pattern 'groups',
> such that you can select all patterns for a group at once. This would be the
> easy button version of the linked multiple expressions to grok reader issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
