Re: Security JIRAS

2014-10-17 Thread Gwen Shapira
Yes, I think we can focus on Broker to Zookeeper communication only. At least for initial stage. Gwen On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino wrote: > For the moment, consumers still need to write under the /consumers tree. > Even if they are committing offsets to Kafka instead of ZK, they

Re: Security JIRAS

2014-10-17 Thread Todd Palino
For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka instead of ZK, they will need to write owner information there when they are balancing. Eventually, you are correct, this is going away with the new consumer. -Todd On Fri, Oct 17

RE: Security JIRAS

2014-10-17 Thread Arvind Mani
I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume "world" read so that consumers don't have to be authenticated (I believe in any case kafka is plann

Re: Security JIRAS

2014-10-16 Thread Gwen Shapira
Wondering the same here :) I think there are some parallel threads here (SSL is independent of Kerberos, as far as I can see). Kerberos work is blocked on https://issues.apache.org/jira/browse/KAFKA-1683 - "Implement a "session" concept in the socket server". So there's no point in picking up oth

Re: Security JIRAS

2014-10-16 Thread Michael Herstine
Thanks, Jay. I¹m new to the project, and I¹m wondering how things proceed from hereŠ are folks working on these tasks, or do they get assigned, orŠ? On 10/7/14, 5:15 PM, "Jay Kreps" wrote: >Hey guys, > >As promised, I added a tree of JIRAs for the stuff in the security wiki ( >https://cwiki.apa

Re: Security JIRAS

2014-10-10 Thread Jarek Jarcec Cecho
I would be strong +1 on that. I’ve seen a lot of regressions on other projects when new functionality cause regressions when running in secure mode. Jarcec On Oct 10, 2014, at 9:43 AM, Neha Narkhede wrote: > I'd vote for accepting every major change with the relevant system tests. > We didn't

Re: Security JIRAS

2014-10-10 Thread Neha Narkhede
I'd vote for accepting every major change with the relevant system tests. We didn't do this for major features in the past that lead to weak coverage and a great deal of work for someone else to add tests for features that were done in the past. I'm guilty of this myself :-( On Thu, Oct 9, 2014 at

Re: Security JIRAS

2014-10-09 Thread Gwen Shapira
Added some details on delegation tokens. I hope it at least clarifies some of the scope. I'm working on a more detailed design doc. On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps wrote: > Hey Gwen, > > Your absolutely right about these. I added the ticket for ZK authentication > and Hadoop delegation

Re: Security JIRAS

2014-10-09 Thread Jarek Jarcec Cecho
> For Kerberos though it isn't clear to me how to do good > integration testing since we need a KDC to test against and it isn't clear > how that happens in the test environment except possibly manually (which is > not ideal). How do other projects handle this? Actually it’s not that hard. Hadoop

Re: Security JIRAS

2014-10-09 Thread Jay Kreps
Hey Gwen, Your absolutely right about these. I added the ticket for ZK authentication and Hadoop delegation tokens. For the Hadoop case I actually don't understand Hadoop security very well. Maybe you could fill in some of the details on what needs to happen for that to work? For testing, we sho

Re: Security JIRAS

2014-10-07 Thread Gwen Shapira
I think we need to add: * Authentication of Kafka brokers with a secured ZooKeeper * Kafka should be able to generate delegation tokens for MapReduce / Spark / Yarn jobs. * Extend systest framework to allow testing secured kafka Gwen On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps wrote: > Hey guys,