Also keep in mind that anything done at the transport (SSL for example) layer won't solve your 'at rest' problems.
All messages are written to disk, so unless the broker does some encryption logic you haven't solved the data visibility issues. I also think this should be a producer/consumer problem not a Broker. Keep the Brokers as fast as possible (thus NIO/kernel space activities etc.) and push the cost to the producers and consumers. Chris On Tue, Apr 23, 2013 at 2:02 PM, Jason Rosenberg <j...@squareup.com> wrote: > Yes, > > I think encryption at the message level is a workable solution, as long as > you don't care about exposing the meta data that goes with it (e.g. topic > names, kafka broker/zk server locations, etc.). > > Jason > > > On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers > <fergal.som...@workday.com>wrote: > > > Hi > > > > We are planning to use Kafka, but like others on this list we have a need > > to be able to secure communication. The approaches people have suggested > on > > this list are: > > > > - Encrypt the messages at the producer (e.g > > > > > http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication > > ) > > - Add SSL to Kafka protocol - > > > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E > > > > Adding SSL support to Kafka, probably means adding SSLEngine support the > > the nio socket handling ( > > https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ > ). > > I don't think there are any immediate plans to provide this, but it's > > potentially something that Kafka would support in the future? > > > > In theory this is something we could look at, but we would need to go > > further. We also need to separate producers from consumers. The aim would > > be to ensure that a Kafka producer couldn't also act as a consumer. > > Essentially producers can write to Kafka, but not read. > > > > From looking at the Kafka source, achieving producer/consumer separation > > looks to me like it would be quite a change to the Kafka server (0.7). So > > are there any plans in the (near) future in this area (producer / > consumer > > separation) ? > > > > Message encryption (at the application layer) would allow us to achieve > > both aims of securing communication and separating consumers from > > producers. Producers get the public cert (so they can encrypt messages as > > they place them on the bus). Only consumers get the private cert - so > only > > they can decrypt messages consumed. This seems like something we can do > > ourselves - I just wanted to sanity check the approach with this group. > > > > Cheers, > > > > Fergal. > > >
