Hi all, after the following Twitter conversation ...
https://twitter.com/jfield/status/715299287479877632 I'd like to explain better my concerns about using Kafka Connect for an AMQP connector. I started to develop it almost one month ago (only on source side, https://github.com/ppatierno/kafka-connect-amqp) but then switched to develop a bridge by myself without using Kafka Connect due to some AMQP needs not well supported by Connect framework. Following the first problems I noticed ... In my implementation, the so called "source connector" creates a ProtonServer to accept AMQP connection from sender client and gives the possibility to the related "source task" to receive message on an AMQP connection, extract address, partition (that should be Kafka Topic and related partition) and body and finally return a structured data injected into Kafka. The first big problem is that if the injected message has a problem (for example the specified partition doesn't exist for the topic), my application logic doesn't receive any exception or any called callback to have such information for handling. This could work fine for pre-settled message (At Most One) but not for At Least One delivery. I need to return a "disposition" message to the client with the delivery state of accepted or rejected. If I want to use the same framework to receive from Kafka but on AMQP connection as well, I have a bigger problem. My current implementation hasn't the receiver side yet but it should be as follow : the "sink connector" has to create a ProtonServer to accept AMQP connection and start reading messages from a Kafka topic that should be the AMQP address (node) specified by the remote AMQP connected receiver. The big problem is that in the Kafka Connect framework architecture, the topic from which to read MUST be defined statically in a configuration file used by the framework itself on startup. In my scenario, I need to start reading from a Kafka topic defined "dinamically" because it depends on the address/node specified by the remote AMQP receiver. The receiver attach to a link on an address that should be the Kafka topic to consume; the address is defined at runtime and the pattern proposal (https://issues.apache.org/jira/browse/KAFKA-3073) can't be enough. Paolo Paolo PatiernoSenior Software Engineer Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperienceBlog : Embedded101
