[
https://issues.apache.org/jira/browse/NIFI-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128757#comment-16128757
]
ASF GitHub Bot commented on NIFI-3484:
--------------------------------------
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/2091
NIFI-3484 GenerateTableFetch Should Allow for Right Boundary
To replace PR #1513
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi PR1513
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2091.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2091
----
commit dde372cf3e4179a4442088adddddbe9775108f7b
Author: patricker <[email protected]>
Date: 2017-02-15T19:11:56Z
NIFI-3484
commit e611456ba71a1ec7fd946fd3290cce2f1610aec0
Author: Pierre Villard <[email protected]>
Date: 2017-08-02T13:52:06Z
fix checkstyle issue, and added unit test showing data duplication issue
----
> GenerateTableFetch Should Allow for Right Boundary
> --------------------------------------------------
>
> Key: NIFI-3484
> URL: https://issues.apache.org/jira/browse/NIFI-3484
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Affects Versions: 1.2.0
> Reporter: Peter Wicks
> Assignee: Peter Wicks
> Priority: Minor
>
> When using GenerateTableFetch it places no right hand boundary on pages of
> data. This can lead to issues when the statement says to get the next 1000
> records greater then a specific key, but records were added to the table
> between the time the processor executed and when the SQL is being executed.
> As a result it pulls in records that did not exist when the processor was
> run. On the next execution of the processor these records will be pulled in
> a second time.
> Example:
> Partition Size = 1000
> First run (no state): Count(*)=4700 and MAX(ID)=4700.
> 5 FlowFiles are generated, the last one will say to fetch 1000, not 700. (But
> I don't think this is really a bug, just an observation).
> 5 Flow Files are now in queue to be executed by ExecuteSQL. Before the 5th
> file can execute 400 new rows are added to the table. When the final SQL
> statement is executed 300 extra records, with higher ID values, will also be
> pulled into NiFi.
> Second run (state: ID=4700). Count(*) ID>4700 = 400 and MAX(ID)=5100.
> 1 Flow File is generated, but includes 300 records already pulled into NiFI.
> The solution is to have an optional property that will let users use the new
> MAX(ID) as a right boundary when generating queries.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
