You might follow along with the Envoy proxy team and the work they are doing to support the Kafka binary protocol: https://github.com/envoyproxy/envoy/issues/2852
On Tue, Mar 19, 2019 at 11:46 AM Peter Bukowinski <pmb...@gmail.com> wrote: > https://docs.confluent.io/3.0.0/kafka-rest/docs/intro.html > > The Kafka REST proxy may be what you need. You can put multiple instances > behind a load balancer to scale to your needs. > > > -- Peter (from phone) > > > On Mar 19, 2019, at 8:30 AM, Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > > Hello James, I'm not aware of anything like that for Kafka, but you can > use > > MirrorMaker for network segmentation. With this approach you have one > Kafka > > cluster in each segment and a MM cluster in the more privileged segment. > > You don't need to expose the privileged segment at all -- you just need > to > > let MM reach the external segment(s). > > > > Ryanne > > > >> On Tue, Mar 19, 2019, 10:20 AM James Grant <ja...@queeg.org> wrote: > >> > >> Hello, > >> > >> We would like to expose a Kafka cluster running on one network to > clients > >> that are running on other networks without having to have full routing > >> between the two networks. In this case these networks are in different > AWS > >> accounts but the concept applies more widely. We would like to access > Kafka > >> over a single (or very few) host names. > >> > >> In addition we would like to filter incoming messages to enforce some > level > >> of data quality and also impose some access control. > >> > >> A solution we are looking into is to provide a Kafka protocol level > proxy > >> that presents to clients as a single node Kafka cluster holding all the > >> topics and partitions of the cluster behind it. This proxy would be > able to > >> operate in a load balanced cluster behind a single DNS entry and would > also > >> be able to intercept and filter/alter messages as they passed through. > >> > >> The advantages we see in this approach over the HTTP proxy is that it > >> presents the Kafka protocol whilst also meaning that we can use a > typical > >> TCP level load balancer that it is easy to route connections to. This > means > >> that we continue to use native Kafka clients. > >> > >> Does anything like this already exist? Does anybody think it would > useful? > >> Does anybody know of any reason it would be impossible (or a bad idea) > to > >> do? > >> > >> James Grant > >> > >> Developer - Expedia Group > >> >
