[
https://issues.apache.org/jira/browse/KAFKA-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Battacharya updated KAFKA-9508:
-------------------------------------
Description:
Kafka connectors which interact with database specially sink connectors need to
know how to handle field length mismatches.
Most databases like oracle enforce field lengths but there is no way to enforce
the same on Avro.
Sample applies with not null fields where records coming in with empty strings
need to be rejected.
We probably can write KSQL or stream jobs to filter out these records but the
KSQL query can get very big and difficult to manage as there might be hundreds
of fields in a table. An easier approach probably will be in the connect put
method to filter out these records using the database metadata already
available to the connector and then either discard these bad records or write
them into a DLQ topic.
was:Kafka connectors which interact with database specially sink connectors
need to know how to handle field length mismatches. Most databases like oracle
enforce field lengths but there is no way to enforce the same on Avro. We
probably can write KSQL or stream jobs to filter out these records but the KSQL
query can get very big and difficult to manage as there might be hundreds of
fields in a table. An easier approach probably will be in the connect put
method to filter out these records using the database metadata already
available to the connector and then either discard these bad records or write
them into a DLQ topic.
> Kafka connect handling of database length checks
> ------------------------------------------------
>
> Key: KAFKA-9508
> URL: https://issues.apache.org/jira/browse/KAFKA-9508
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 1.1.0
> Reporter: Rahul Battacharya
> Priority: Major
>
> Kafka connectors which interact with database specially sink connectors need
> to know how to handle field length mismatches.
> Most databases like oracle enforce field lengths but there is no way to
> enforce the same on Avro.
> Sample applies with not null fields where records coming in with empty
> strings need to be rejected.
> We probably can write KSQL or stream jobs to filter out these records but the
> KSQL query can get very big and difficult to manage as there might be
> hundreds of fields in a table. An easier approach probably will be in the
> connect put method to filter out these records using the database metadata
> already available to the connector and then either discard these bad records
> or write them into a DLQ topic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
