Kafka Connect was designed explicitly to support this kind of use case. Debezium's connectors implement the Kafka Connect API, and they just released a MS SQL Server source connector that captures the changes from SQL Server and writes them to Kafka. Debezium's MySQL source connector may work with MariaDB, but it's probably best to check with their community () about that.
There are other sink connectors for reading Kafka topics and writing to S3, including Confluent's S3 sink connector ( https://docs.confluent.io/current/connect/kafka-connect-s3/index.html#connect-s3 ). Of course the connector's I've mentioned may not be the only options for the systems you mentioned, and there are lots of source and sink connectors for other systems. Best regards, Randall On Thu, Feb 14, 2019 at 12:03 PM John Ziebro <jzie...@mbsicorp.com> wrote: > Hi, > > We are planning on implementing Kafka to stream changes from a MariaDB as > well as a MS SQL Server database into an S3 bucket and would like to stay > with opensource solutions as much as possible. To simplify this process it > looks like we might need to use something that leverages Kafka like > Debezium<https://debezium.io/>. > > What are your suggestions for handling this scenario? > > Thank you, > John Z > >
