rbaumgar opened a new issue #338:
URL: https://github.com/apache/camel-kafka-connector/issues/338
try to use CamelAws2s3SourceConnector to connect to another S3 provider
(minio)
when I tyr to upload a simple text file to my s2 bucket -> error
KafkaConnector
============
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: aws2-s3-source-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
config:
camel.component.aws2-s3.configuration.access-key: minio
camel.component.aws2-s3.configuration.secret-key: minio123
camel.component.aws2-s3.configuration.region: EU_WEST_1
camel.source.endpoint.autocloseBody: false
camel.source.endpoint.overrideEndpoint: true
camel.source.endpoint.uriEndpointOverride: http://minio-server
camel.source.maxPollDuration: 10000
camel.source.path.bucketNameOrArn: camel-connector-test
key.converter: org.apache.kafka.connect.storage.StringConverter
topics: s3-topic
value.converter:
org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter
tasksMax: 1
Log
===
2020-07-28 12:35:44,903 INFO Setting initial properties in Camel context:
[{camel.source.endpoint.autocloseBody=false,
connector.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector,
camel.source.maxPollDuration=5000,
camel.source.endpoint.runLoggingLevel=TRACE,
camel.component.aws2-s3.configuration.region=EU_WEST_1, topics=s3-topic,
tasks.max=1, camel.source.endpoint.overrideEndpoint=true,
camel.source.endpoint.uriEndpointOverride=http://minio-server,
camel.source.component=aws2-s3,
task.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceTask,
camel.source.path.bucketNameOrArn=camel-connector-test,
camel.component.aws2-s3.configuration.access-key=minio,
name=aws2-s3-source-connector,
camel.component.aws2-s3.configuration.secret-key=minio123,
value.converter=org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter,
key.converter=org.apache.kafka.connect.storage.StringConverter}]
(org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task
-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,913 INFO Creating Camel route
from(aws2-s3://camel-connector-test?autocloseBody=false&overrideEndpoint=true&runLoggingLevel=TRACE&uriEndpointOverride=http://minio-server).to(direct:end?pollingConsumerQueueSize=1000&pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true)
(org.apache.camel.kafkaconnector.utils.CamelMainSupport)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,916 INFO [Producer
clientId=connector-producer-aws2-s3-source-connector-0] Cluster ID:
k_EXjSBzQvmhHkadWFBbcg (org.apache.kafka.clients.Metadata)
[kafka-producer-network-thread | connector-producer-aws2-s3-source-connector-0]
2020-07-28 12:35:44,922 INFO Starting CamelContext
(org.apache.camel.kafkaconnector.utils.CamelMainSupport)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,923 INFO Using properties from:
classpath:application.properties;optional=true
(org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO Auto-configuration summary:
(org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO
camel.component.aws2-s3.configuration.region=EU_WEST_1
(org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO
camel.component.aws2-s3.configuration.accessKey=minio
(org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO
camel.component.aws2-s3.configuration.secretKey=minio123
(org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,929 INFO No additional Camel XML routes discovered from:
classpath:camel/*.xml (org.apache.camel.main.DefaultRoutesCollector)
[pool-14-thread-1]
2020-07-28 12:35:44,929 INFO No additional Camel XML rests discovered from:
classpath:camel-rest/*.xml (org.apache.camel.main.DefaultRoutesCollector)
[pool-14-thread-1]
2020-07-28 12:35:44,938 INFO Apache Camel 3.3.0 (CamelContext: camel-4) is
starting (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,938 INFO StreamCaching is not in use. If using streams
then its recommended to enable stream caching. See more details at
http://camel.apache.org/stream-caching.html
(org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Route: route4 started and consuming from:
aws2-s3://camel-connector-test
(org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Total 1 routes, of which 1 are started
(org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Apache Camel 3.3.0 (CamelContext: camel-4)
started in 0.014 seconds (org.apache.camel.impl.engine.AbstractCamelContext)
[pool-14-thread-1]
2020-07-28 12:35:44,953 INFO CamelContext started
(org.apache.camel.kafkaconnector.utils.CamelMainSupport)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,953 INFO CamelSourceTask connector task started
(org.apache.camel.kafkaconnector.CamelSourceTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,953 INFO WorkerSourceTask{id=aws2-s3-source-connector-0}
Source task finished initialization and start
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,968 INFO WorkerSourceTask{id=aws2-s3-source-connector-0}
Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO WorkerSourceTask{id=aws2-s3-source-connector-0}
flushing 0 outstanding messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 ERROR
WorkerSourceTask{id=aws2-s3-source-connector-0} Task threw an uncaught and
unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
[task-thread-aws2-s3-source-connector-0]
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in
error handler
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
at
org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:295)
at
org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:245)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
software.amazon.awssdk.core.ResponseInputStream
at
org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter.fromConnectData(S3ObjectConverter.java:37)
at
org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
at
org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:295)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
... 11 more
2020-07-28 12:35:45,969 ERROR
WorkerSourceTask{id=aws2-s3-source-connector-0} Task is being killed and will
not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Stopping CamelSourceTask connector task
(org.apache.camel.kafkaconnector.CamelSourceTask)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Stopping CamelContext
(org.apache.camel.kafkaconnector.utils.CamelMainSupport)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Apache Camel 3.3.0 (CamelContext: camel-4) is
shutting down (org.apache.camel.impl.engine.AbstractCamelContext)
[task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Starting to graceful shutdown 1 routes (timeout
45 seconds) (org.apache.camel.impl.engine.DefaultShutdownStrategy)
[task-thread-aws2-s3-source-connector-0]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
