[
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abdullah Aqeeli updated NIFI-9477:
----------------------------------
Description:
hello all,
We are using Nifi to do an incremental load from SQL Server using
QueryDatabaseTableRecord where max-value column is updated_at column with a
[Datetime2
|https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
type. The source tables get updated in milliseconds. We noticed that we are
losing few events with each QueryDatabaseTableRecord run. As a work-around we
created a custom processor that subtracts few seconds from the max-value in the
state fetching overlapping events and then we handle any duplicates later on in
the pipeline and it seems to solve the issue. However, this is not optimal.
We'd like to know what's causing the skips. Our assumption so far is that
timestamp datatype accuracy is causing the skips in the incremental load
however we're not sure.
Thanks!
was:
hello all,
We are using Nifi to do an incremental load from SQL Server using
QueryDatabaseTableRecord where max-value column is updated_at column with a
[Datetime2
|https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
type. The source tables get updated in milliseconds. We noticed that we are
losing few events doing with each QueryDatabaseTableRecord run. As a
work-around we created a custom processor that subtracts few seconds from the
max-value in the state fetching overlapping events and then we handle any
duplicates later on in the pipeline and it seems to solve the issue. However,
this is not optimal. We'd like to know what's causing the skips. Our assumption
so far is that timestamp datatype accuracy is causing the skips in the
incremental load however we're not sure.
Thanks!
> data loss doing Incremental load based on a timestamp from SQL server using
> QueryDatabaseTableRecord
> -----------------------------------------------------------------------------------------------------
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: ubuntu 20
> 2-node cluster using an embedded zk
> Reporter: Abdullah Aqeeli
> Priority: Critical
>
> hello all,
>
> We are using Nifi to do an incremental load from SQL Server using
> QueryDatabaseTableRecord where max-value column is updated_at column with a
> [Datetime2
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
> type. The source tables get updated in milliseconds. We noticed that we are
> losing few events with each QueryDatabaseTableRecord run. As a work-around we
> created a custom processor that subtracts few seconds from the max-value in
> the state fetching overlapping events and then we handle any duplicates later
> on in the pipeline and it seems to solve the issue. However, this is not
> optimal. We'd like to know what's causing the skips. Our assumption so far is
> that timestamp datatype accuracy is causing the skips in the incremental load
> however we're not sure.
>
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
