Nick, There will be drawbacks to this because you would be using the wrong tool for the task. The Kafka ecosystem is very rich with different tools and you should spend a little more time to understand what they do and are meant for
I believe the ecosystem component you are looking for is Kafka Streams. You can also use KSQLDB for this if it is simple enough and you want to avoid writing Java/Scala code You should spend some time to understand each of the components of the ecosystem and it will become clearer how they can be leveraged for stream processing: https://kafka.apache.org/documentation/#api Kafka Brokers Kafka Connect Kafka Streams Kafka Producers Kafka Consumers Kafka Admin Clients Confluent Schema Registry Confluent REST Proxy Confluent KSQLDB Take a look at all of the above and try to understand how they are inter connected, dependent and used and it will bring better clarity To directly answer your question, you do not need a connector to process your events, you need to create a Kafka Streams app or depending on the use case leverage KSQLDB for the task. The number of instances can scale up or down as needed. The connect framework was not intended for what you are attempting to use it for. Please follow up with any additional questions you may have I hope this was helpful On Thu, Mar 11, 2021 at 1:27 PM Nick Siviglia <nisivig...@gmail.com> wrote: > Hi Everyone, > > I'd like to create a Kafka connector that instead of acting like a source > or sink will instead do some processing on the data and output to another > kafka topic. Has anyone done this before? Does anyone see any potential > drawbacks? > > Data is json format at around 150 string and number fields per object. And > I'm planning on receiving about 2 million a day. > > Thanks for any help, > Nick >
