shubham-maheshwari-tech opened a new issue, #1682:
URL: https://github.com/apache/camel-kafka-connector/issues/1682
Hi
I am using Apache camel S3 source connector, with Strimizi Operator. When
ever the kafka connector is restarted after some config update duplicate events
are coming in Kafka means older payload which are present in S3 bucket that can
also be seen.
Below is full connector configuration:
`CamelAwss3sourceSourceConnectorConfig values:
camel.aggregation.size = 10
camel.aggregation.timeout = 500
camel.beans.aggregate = null
camel.error.handler = default
camel.error.handler.max.redeliveries = 0
camel.error.handler.redelivery.delay = 1000
camel.idempotency.enabled = true
camel.idempotency.expression.header = null
camel.idempotency.expression.type = body
camel.idempotency.kafka.bootstrap.servers = localhost:9092
camel.idempotency.kafka.max.cache.size = 1000
camel.idempotency.kafka.poll.duration.ms = 100
camel.idempotency.kafka.topic = kafka_idempotent_repository
camel.idempotency.memory.dimension = 100
camel.idempotency.repository.type = memory
camel.kamelet.aws-s3-source.accessKey = [hidden]
camel.kamelet.aws-s3-source.autoCreateBucket = false
camel.kamelet.aws-s3-source.bucketNameOrArn = test-bucket
camel.kamelet.aws-s3-source.delay = 500
camel.kamelet.aws-s3-source.deleteAfterRead = false
camel.kamelet.aws-s3-source.forcePathStyle = false
camel.kamelet.aws-s3-source.ignoreBody = false
camel.kamelet.aws-s3-source.maxMessagesPerPoll = 10
camel.kamelet.aws-s3-source.overrideEndpoint = false
camel.kamelet.aws-s3-source.prefix = schema/feature
camel.kamelet.aws-s3-source.region = ap-southeast-1
camel.kamelet.aws-s3-source.secretKey = [hidden]
camel.kamelet.aws-s3-source.uriEndpointOverride = null
camel.kamelet.aws-s3-source.useDefaultCredentialsProvider = false
camel.map.headers = true
camel.map.properties = true
camel.remove.headers.pattern =
camel.source.camelMessageHeaderKey = CamelAwsS3Key
camel.source.component = null
camel.source.contentLogLevel = OFF
camel.source.marshal = null
camel.source.maxBatchPollSize = 1000
camel.source.maxNotCommittedRecords = 1024
camel.source.maxPollDuration = 1000
camel.source.pollingConsumerBlockTimeout = 0
camel.source.pollingConsumerBlockWhenFull = true
camel.source.pollingConsumerQueueSize = 1000
camel.source.unmarshal = null
camel.source.url = null
topics = commerce-test
`
The **camel.idempotency.enabled = true** is already enabled but still
duplicate events coming on every kafka connector restart.
Using 4.4.2 version apache camel lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
