Eric, this idea came up multiple time already, but it is much more difficult to build than it seems, especially for supporting "exactly once". Kafka transaction only work on a single cluster...
Maybe we will support it at some point, but the "workaround" to either replicate the input or output topic(s) is fairly simple and thus it seems not to be a "killer feature" but more like a "nice to have" (just my personal opinion). -Matthias On 12/1/20 2:08 PM, Malcolm McFarland wrote: > How about a lightweight MirrorMaker instance for just this topic? > > Malcolm McFarland > Cavulus > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > unauthorized or improper disclosure, copying, distribution, or use of the > contents of this message is prohibited. The information contained in this > message is intended only for the personal and confidential use of the > recipient(s) named above. If you have received this message in error, > please notify the sender immediately and delete the original message. > > > On Tue, Dec 1, 2020 at 1:18 PM Eric Beabes <mailinglist...@gmail.com> wrote: > >> Hmm.. Ok. Maybe this could be a feature request for a future release. This >> can be accomplished easily in Spark Structured Streaming... just saying :) >> >> In Spark Structured Streaming we can have separate configurations for >> 'readStream' & 'writeStream'. I am a bit surprised this is not available in >> Kafka Streams. >> >> Even during development sometimes we want to read from a topic in a QA >> cluster & write it to a topic on the Local cluster. >> >> >> >> On Tue, Dec 1, 2020 at 11:33 AM Matthias J. Sax <mj...@apache.org> wrote: >> >>> KafkaStreams can only connect to a single cluster. >>> >>> If you really need to read from one cluster and write to another, you >>> have 3 main options: >>> >>> - use KafkaStreams on the source cluster and mirror the output topic >>> from the source to the target cluster >>> >>> - mirror the input topic from the source cluster to the target cluster >>> and use KafkaStreams on the target cluster >>> >>> - don't use KafkaStreams but plain consumer/producer >>> >>> >>> >>> -Matthias >>> >>> On 12/1/20 10:58 AM, Eric Beabes wrote: >>>> I need to read from a topic in one bootstrap server & write it to >> another >>>> topic in another bootstrap server. Since there's only one >>>> StreamsConfig.BOOTSTRAP_SERVERS_CONFIG property, I am wondering how to >>>> accomplish this? >>>> >>>> Do I need to create 2 different KafkaStreams objects? One for reading & >>> the >>>> other for writing? >>>> >>> >> >
