[
https://issues.apache.org/jira/browse/NIFI-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847844#comment-15847844
]
ASF GitHub Bot commented on NIFI-2881:
--------------------------------------
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1407#discussion_r98810570
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
---
@@ -212,9 +215,21 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
final Map<String, String> statePropertyMap = new
HashMap<>(stateMap.toMap());
//If an initial max value for column(s) has been specified using
properties, and this column is not in the state manager, sync them to the state
property map
- for(final Map.Entry<String,String> maxProp :
maxValueProperties.entrySet()){
- if
(!statePropertyMap.containsKey(maxProp.getKey().toLowerCase())) {
- statePropertyMap.put(maxProp.getKey().toLowerCase(),
maxProp.getValue());
+ for (final Map.Entry<String, String> maxProp :
maxValueProperties.entrySet()) {
+ String maxPropKey = maxProp.getKey().toLowerCase();
+ String fullyQualifiedMaxPropKey = getStateKey(tableName,
maxPropKey);
+ if (!statePropertyMap.containsKey(fullyQualifiedMaxPropKey)) {
+ String newMaxPropValue;
+ // If we can't find the value at the fully-qualified key
name, it is possible (under a previous scheme)
+ // the value has been stored under a key that is only the
column name. Fall back to check the column name,
+ // but store the new initial max value under the
fully-qualified key.
+ if (statePropertyMap.containsKey(maxPropKey)) {
+ newMaxPropValue = statePropertyMap.get(maxPropKey);
+ } else {
+ newMaxPropValue = maxProp.getValue();
--- End diff --
I was suggesting that for QueryDatabaseTable and Initial max value, instead
of GenerateTableFetch. Because not being able to know the column type seemed an
obstacle to support incoming Flow Files in QueryDatabaseTable ([PR
comment](https://github.com/apache/nifi/pull/1407#issuecomment-274592165) and
[JIRA
comment](https://issues.apache.org/jira/browse/NIFI-2881?focusedCommentId=15811850&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15811850)).
However, I agree with you that, as you've written, since GenerateTableFetch
and ExecuteSQL can be used if user need to pass arguments from incoming flow
files, it's not necessary to support incoming flow files in QueryDatabaseTable.
If we are not going to add incoming flow file support for
QueryDatabaseTable in this PR, maybe we should add some docs in
QueryDatabaseTable capability description to guide users to refer
GenerateTableFetch and ExecuteSQL, too.
> Allow Database Fetch processor(s) to accept incoming flow files and use
> Expression Language
> -------------------------------------------------------------------------------------------
>
> Key: NIFI-2881
> URL: https://issues.apache.org/jira/browse/NIFI-2881
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
>
> The QueryDatabaseTable and GenerateTableFetch processors do not allow
> Expression Language to be used in the properties, mainly because they also do
> not allow incoming connections. This means if the user desires to fetch from
> multiple tables, they currently need one instance of the processor for each
> table, and those table names must be hard-coded.
> To support the same capabilities for multiple tables and more flexible
> configuration via Expression Language, these processors should have
> properties that accept Expression Language, and GenerateTableFetch should
> accept (optional) incoming connections.
> Conversation about the behavior of the processors is welcomed and encouraged.
> For example, if an incoming flow file is available, do we also still run the
> incremental fetch logic for tables that aren't specified by this flow file,
> or do we just do incremental fetching when the processor is scheduled but
> there is no incoming flow file. The latter implies a denial-of-service could
> take place, by flooding the processor with flow files and not letting it do
> its original job of querying the table, keeping track of maximum values, etc.
> This is likely a breaking change to the processors because of how state
> management is implemented. Currently since the table name is hard coded, only
> the column name comprises the key in the state. This would have to be
> extended to have a compound key that represents table name, max-value column
> name, etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
