The other issue with this model is that you're Kafka servers are available on the internet ... so anyone else can push data into them. At least, if you are running in a cross-cloud/cross-datacenter environment. We're struggling with the same design issues right now.
What we've hoped for is that in 0.8.x, Kafka would allow our producers to connect to ANY Kafka server and submit data, and have that data dynamically routed to the right servers. This way we could put the Kafka servers behind an ELB, throw up Stunnel clients on our producers and Stunnel servers on the Kafka machines. This would offload the SSL encryption and authentication to Stunnel, and allow Kafka to concentrate on what its good at. For now though, that doesn't seem possible. It looks like we may end up going down the Flume route, because its easier to encrypt and authenticate the data streams through Flume. :/ --Matt On Apr 23, 2013, at 11:02 AM, Jason Rosenberg <j...@squareup.com> wrote: > Yes, > > I think encryption at the message level is a workable solution, as long as > you don't care about exposing the meta data that goes with it (e.g. topic > names, kafka broker/zk server locations, etc.). > > Jason > > > On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers > <fergal.som...@workday.com>wrote: > >> Hi >> >> We are planning to use Kafka, but like others on this list we have a need >> to be able to secure communication. The approaches people have suggested on >> this list are: >> >> - Encrypt the messages at the producer (e.g >> >> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication >> ) >> - Add SSL to Kafka protocol - >> >> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E >> >> Adding SSL support to Kafka, probably means adding SSLEngine support the >> the nio socket handling ( >> https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ). >> I don't think there are any immediate plans to provide this, but it's >> potentially something that Kafka would support in the future? >> >> In theory this is something we could look at, but we would need to go >> further. We also need to separate producers from consumers. The aim would >> be to ensure that a Kafka producer couldn't also act as a consumer. >> Essentially producers can write to Kafka, but not read. >> >> From looking at the Kafka source, achieving producer/consumer separation >> looks to me like it would be quite a change to the Kafka server (0.7). So >> are there any plans in the (near) future in this area (producer / consumer >> separation) ? >> >> Message encryption (at the application layer) would allow us to achieve >> both aims of securing communication and separating consumers from >> producers. Producers get the public cert (so they can encrypt messages as >> they place them on the bus). Only consumers get the private cert - so only >> they can decrypt messages consumed. This seems like something we can do >> ourselves - I just wanted to sanity check the approach with this group. >> >> Cheers, >> >> Fergal. >>
