Re: kafka-connect-jdbc: ids, timestamps, and transactions

2015-12-18 Thread James Cheng
Mark, what database are you using? If you are using MySQL... There is a not-yet-finished Kafka MySQL Connector at https://github.com/wushujames/kafka-mysql-connector. It tails the MySQL binlog, and so will handle the situation you describe. But, as I mentioned, I haven't finished it yet. If

Re: kafka-connect-jdbc: ids, timestamps, and transactions

2015-12-18 Thread Mark Drago
Ewen, Thanks for the reply. We'll proceed while keeping all of your points in mind. I looked around for a more focused forum for the jdbc connector before posting here but didn't come across the confluent-platform group. I'll direct any more questions about the jdbc connector there. I'll also c

Re: kafka-connect-jdbc: ids, timestamps, and transactions

2015-12-16 Thread Ewen Cheslack-Postava
Mark, There are definitely limitations to using JDBC for change data capture. Using a database-specific implementation, especially if you can read directly off the database's log, will be able to handle more situations like this. Cases like the one you describe are difficult to address efficiently