Yes securing the content of log messages at rest is important to us - which favors message encryption. Thanks for the responses.
Fergal. On Tue, Apr 23, 2013 at 7:31 PM, Chris Curtin <curtin.ch...@gmail.com>wrote: > Also keep in mind that anything done at the transport (SSL for example) > layer won't solve your 'at rest' problems. > > All messages are written to disk, so unless the broker does some encryption > logic you haven't solved the data visibility issues. > > I also think this should be a producer/consumer problem not a Broker. Keep > the Brokers as fast as possible (thus NIO/kernel space activities etc.) and > push the cost to the producers and consumers. > > Chris > > > On Tue, Apr 23, 2013 at 2:02 PM, Jason Rosenberg <j...@squareup.com> wrote: > > > Yes, > > > > I think encryption at the message level is a workable solution, as long > as > > you don't care about exposing the meta data that goes with it (e.g. topic > > names, kafka broker/zk server locations, etc.). > > > > Jason > > > > > > On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers > > <fergal.som...@workday.com>wrote: > > > > > Hi > > > > > > We are planning to use Kafka, but like others on this list we have a > need > > > to be able to secure communication. The approaches people have > suggested > > on > > > this list are: > > > > > > - Encrypt the messages at the producer (e.g > > > > > > > > > http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication > > > ) > > > - Add SSL to Kafka protocol - > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E > > > > > > Adding SSL support to Kafka, probably means adding SSLEngine support > the > > > the nio socket handling ( > > > > https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ > > ). > > > I don't think there are any immediate plans to provide this, but it's > > > potentially something that Kafka would support in the future? > > > > > > In theory this is something we could look at, but we would need to go > > > further. We also need to separate producers from consumers. The aim > would > > > be to ensure that a Kafka producer couldn't also act as a consumer. > > > Essentially producers can write to Kafka, but not read. > > > > > > From looking at the Kafka source, achieving producer/consumer > separation > > > looks to me like it would be quite a change to the Kafka server (0.7). > So > > > are there any plans in the (near) future in this area (producer / > > consumer > > > separation) ? > > > > > > Message encryption (at the application layer) would allow us to achieve > > > both aims of securing communication and separating consumers from > > > producers. Producers get the public cert (so they can encrypt messages > as > > > they place them on the bus). Only consumers get the private cert - so > > only > > > they can decrypt messages consumed. This seems like something we can do > > > ourselves - I just wanted to sanity check the approach with this group. > > > > > > Cheers, > > > > > > Fergal. > > > > > > -- Fergal Somers Architect Integrations (Skype) fergal.somers.workday fergal.som...@workday.com <http://www.workday.com/signature-uk>
