Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Gary, I have used this technique before. I deleted flink-avro jar from lib and packed it into the application jar and there are no problems. Best, Nick On Thu, May 14, 2020 at 6:11 AM Gary Yao wrote: > Its because the flink distribution of the cluster is 1.7.2. We use a >> standalone cluste

Re: ProducerRecord with Kafka Sink for 1.8.0

> > Its because the flink distribution of the cluster is 1.7.2. We use a > standalone cluster , so in the lib directory in flink the artifact is > flink-core-1.7.2.jar . I need to pack flink-core-1.9.0.jar from application > and use child first class loading to use newer version of flink-core. > D

Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Gary, Its because the flink distribution of the cluster is 1.7.2. We use a standalone cluster , so in the lib directory in flink the artifact is flink-core-1.7.2.jar . I need to pack flink-core-1.9.0.jar from application and use child first class loading to use newer version of flink-core. If I

Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Nick, Can you explain why it is required to package flink-core into your application jar? Usually flink-core is a dependency with provided scope [1] Best, Gary [1] https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#dependency-scope On Tue, May 12, 2020 at

Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Gary, Thanks for the info. I am aware this feature is available in 1.9.0 onwards. Our cluster is still very old and have CICD challenges,I was hoping not to bloat up the application jar by packaging even flink-core with it. If its not possible to do this with older version without writing our ow

Re: ProducerRecord with Kafka Sink for 1.8.0

Hi Nick, Are you able to upgrade to Flink 1.9? Beginning with Flink 1.9 you can use KafkaSerializationSchema to produce a ProducerRecord [1][2]. Best, Gary [1] https://issues.apache.org/jira/browse/FLINK-11693 [2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/api/java/org/apache/f

ProducerRecord with Kafka Sink for 1.8.0

Hi guys, I use 1.8.0 version for flink-connector-kafka. Do you have any recommendations on how to produce a ProducerRecord from a kafka sink. Looking to add support to kafka headers therefore thinking about ProducerRecord. If you have any thoughts its highly appreciated. Best, Nick.