[
https://issues.apache.org/jira/browse/NIFI-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826349#comment-15826349
]
ASF GitHub Bot commented on NIFI-2881:
--------------------------------------
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1407
If we going to add the ability to keep track of dynamic max-value
properties in GenerateTableFetch, then there's really no reason why we can't do
the same for QueryDatabaseTable. I left it out of the original Jira based on
the premise that we wouldn't support max-value columns with incoming
connections, and at that point QueryDatabaseTable would work basically like
ExecuteSQL.
The major difference in this case will be that if there is an incoming
connection, then the fetch processors are purely event-driven, meaning they
will need to be hydrated from upstream to get new results. If there is no
incoming connection, then they will maintain their current behavior of running
on their current schedule to check for new max values. Does this sound ok?
> Allow Database Fetch processor(s) to accept incoming flow files and use
> Expression Language
> -------------------------------------------------------------------------------------------
>
> Key: NIFI-2881
> URL: https://issues.apache.org/jira/browse/NIFI-2881
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
>
> The QueryDatabaseTable and GenerateTableFetch processors do not allow
> Expression Language to be used in the properties, mainly because they also do
> not allow incoming connections. This means if the user desires to fetch from
> multiple tables, they currently need one instance of the processor for each
> table, and those table names must be hard-coded.
> To support the same capabilities for multiple tables and more flexible
> configuration via Expression Language, these processors should have
> properties that accept Expression Language, and GenerateTableFetch should
> accept (optional) incoming connections.
> Conversation about the behavior of the processors is welcomed and encouraged.
> For example, if an incoming flow file is available, do we also still run the
> incremental fetch logic for tables that aren't specified by this flow file,
> or do we just do incremental fetching when the processor is scheduled but
> there is no incoming flow file. The latter implies a denial-of-service could
> take place, by flooding the processor with flow files and not letting it do
> its original job of querying the table, keeping track of maximum values, etc.
> This is likely a breaking change to the processors because of how state
> management is implemented. Currently since the table name is hard coded, only
> the column name comprises the key in the state. This would have to be
> extended to have a compound key that represents table name, max-value column
> name, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
