[
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Aqeeli updated NIFI-9477:
----------------------------------
Component/s: Core Framework
(was: Core UI)
> data loss doing Incremental load based on a timestamp from SQL server using
> QueryDatabaseTableRecord
> -----------------------------------------------------------------------------------------------------
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.13.2
> Environment: ubuntu 20
> 2-node cluster using an embedded zk
> Reporter: Abdullah Aqeeli
> Priority: Critical
>
> hello all,
>
> We are using Nifi to do an incremental load from SQL Server using
> QueryDatabaseTableRecord where max-value column is updated_at column with a
> [Datetime2
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
> type. The source tables get updated in milliseconds. We noticed that we are
> losing few events doing with each QueryDatabaseTableRecord run. As a
> work-around we created a custom processor that subtracts few seconds from the
> max-value in the state fetching overlapping events and then we handle any
> duplicates later on in the pipeline and it seems to solve the issue. However,
> this is not optimal. We'd like to know what's causing the skips. Our
> assumption so far is that timestamp datatype accuracy is causing the skips in
> the incremental load however we're not sure.
>
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
