Re: Kafka Connectors output to topic.

Nick, There will be drawbacks to this because you would be using the wrong tool for the task. The Kafka ecosystem is very rich with different tools and you should spend a little more time to understand what they do and are meant for I believe the ecosystem component you are looking for is Kafka S

Re: Kafka Connectors output to topic.

Hi Nick, Have you taken this further? For example, have you considered using Spark Structured Streaming as an effective ETL tool here. You want to do some ETL then, you can define your micro-batch to get your topic ingested from Kafka upstream, write your own transformation in your language of c

Re: Kafka Connectors output to topic.

For what it's worth, that sounds less like a Kafka Connector and more like a Kafka Streams app. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 11 Mar 2021 at 18:27, Nick Siviglia wrote: > Hi Everyone, > > I'd like to create a Kafka connector that instead

Re: Kafka Connectors output to topic.

so you want to do ETL on the input data Kafka source (topic 1) -> Processing -> Another Kafka sink (topic2) LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Dis

Kafka Connectors output to topic.

Hi Everyone, I'd like to create a Kafka connector that instead of acting like a source or sink will instead do some processing on the data and output to another kafka topic. Has anyone done this before? Does anyone see any potential drawbacks? Data is json format at around 150 string and number f