Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-28 Thread John Roesler
Hi Paul, No problem! And please let us know how it goes. Thanks, -John On Wed, Mar 27, 2019 at 9:13 PM Paul Whalen wrote: > John, > > You make a good case for it already being a public API, so my nerves are > definitely eased on that front. I do think we have a path to move forward > with the

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-27 Thread Paul Whalen
John, You make a good case for it already being a public API, so my nerves are definitely eased on that front. I do think we have a path to move forward with the user space solution, and if I get a chance, I'm going to try proving it out with a trivial demo using an early MM2 build - but it's nice

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-27 Thread John Roesler
Hi Paul, Sorry for overlooking the "offset translation" MM2 feature. I'm glad Ryanne was able to confirm this would work. I'm just one voice, but FWIW, I think that the internal topic naming scheme is a public API. We document the structure of the naming scheme in several places. We also recommen

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-25 Thread Paul Whalen
John and Ryanne, Thanks for the responses! I think Ryanne's way of describing the question is actually a much better summary than my long winded description: "a Streams app can switch between topics with and without a cluster alias prefix when you migrate between prod and pre-prod, while preservin

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-25 Thread Ryanne Dolan
Hey Paul, thanks for the kind words re MM2. I'm not a Streams expert first off, but I think I understand your question: if a Streams app can switch between topics with and without a cluster alias prefix when you migrate between prod and pre-prod, while preserving state. Streams supports regexes an

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-25 Thread John Roesler
Hi Paul, Thanks for the email. This does seem like a good setup to support. This might seem a little low-fi, but do you think it would work to handle this use case entirely in "user space"? I may be missing something because this is off the cuff... In the code for your Streams app, I'm wondering

MirrorMaker 2.0 and Streams interplay (topic naming control)

2019-03-24 Thread Paul Whalen
Hi all, With MirrorMaker 2.0 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0) accepted and coming along very nicely in development, it has got me wondering if a certain use case is supported, and if not, can changes be made to Streams or MM2 to support it. I'll expl