[VOTE] KIP-123: Allow per stream/table timestamp extractor

[Jeyhun Karimov]

Dear community,

I'd like to start the vote for KIP-123:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788


Cheers,
Jeyhun
-- 
-Cheers

Jeyhun

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Damian Guy]

Thanks for the KIP Jeyhun!

+1

On Tue, 28 Feb 2017 at 08:59 Jeyhun Karimov  wrote:

> Dear community,
>
> I'd like to start the vote for KIP-123:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
>
>
> Cheers,
> Jeyhun
> --
> -Cheers
>
> Jeyhun
>

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Damian Guy]

+1

On Tue, 28 Feb 2017 at 04:32 Vahid S Hashemian 
wrote:

> +1 on 0.11.0.0.
>
> Can we also include KIP-54 to the list?
> The PR for this KIP is ready for review.
>
> Thanks.
> --Vahid
>
>
>
>
>
> From:   Ismael Juma 
> To: dev@kafka.apache.org
> Date:   02/27/2017 07:47 PM
> Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> Sent by:isma...@gmail.com
>
>
>
> Hi all,
>
> With 0.10.2.0 out of the way, I would like to volunteer to be the release
> manager for our next time-based release. See https://cwiki.apache.org/c
> onfluence/display/KAFKA/Time+Based+Release+Plan
> 
> if you missed previous
> communication on time-based releases or need a reminder.
>
> I put together a draft release plan with June 2017 as the release month
> (as
> previously agreed) and a list of KIPs that have already been voted:
>
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68716876
>  >*
>
> I haven't set exact dates for the various stages (feature freeze, code
> freeze, etc.) for now as Ewen is going to send out an email with some
> suggested tweaks based on his experience as release manager for 0.10.2.0.
> We can set the exact dates after that discussion.
>
> As we are starting the process early this time, we should expect the
> number
> of KIPs in the plan to grow (so don't worry if your KIP is not there yet),
> but it's good to see that we already have 10 (including 2 merged and 2
> with
> PR reviews in progress).
>
> Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and KIP-101
> (Leader Generation in Replication) require message format changes, which
> typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> it makes sense to also include KIP-106 (Unclean leader election should be
> false by default) and KIP-118 (Drop support for Java 7). We would also
> take
> the chance to remove deprecated code, in that case.
>
> Given the above, how do people feel about 0.11.0.0 as the next Kafka
> version? Please share your thoughts.
>
> Thanks,
> Ismael
>
>
>
>
>

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

[Eno Thereska]

Hi Jeyhun,

Thanks for the KIP, sorry I'm coming a bit late to the discussion.

One thing I'd like to understand is whether we can avoid situations where the 
user is mixing different times (event time vs. wallclock time) in their 
processing inadvertently. Before this KIP, all the relevant topics have one 
time stamp extractor so that issue does not come up.

What will be the behavior if times mismatch, e.g., for joins?

Thanks
Eno

> On 22 Feb 2017, at 09:21, Jeyhun Karimov  wrote:
> 
> Dear community,
> 
> I would like to get further feedbacks on this KIP (if any).
> 
> Cheers
> Jeyhun
> 
> On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax 
> wrote:
> 
>> Mathieu,
>> 
>> I personally agree with your observation, and we have plans to submit a
>> KIP like this. If you want to drive this discussion feel free to start
>> the KIP by yourself!
>> 
>> Having said that, for this KIP we might want to focus the discussion the
>> the actual feature that gets added: allowing to specify different
>> TS-Extractor for different inputs.
>> 
>> 
>> 
>> -Matthias
>> 
>> On 2/14/17 4:54 PM, Mathieu Fenniak wrote:
>>> Hi Jeyhun,
>>> 
>>> This KIP might not be the appropriate time, but my first thought reading
>> it
>>> is that it might make sense to introduce a builder-style API rather than
>>> adding a mix of new method overloads with independent optional
>> parameters.
>>> :-)
>>> 
>>> eg. stream(), table(), globalTable(), addSource(), could all accept a
>>> "TopicReference" parameter that can be built like:
>>> 
>> TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build().
>>> 
>>> Mathieu
>>> 
>>> 
>>> On Tue, Feb 14, 2017 at 5:31 PM, Jeyhun Karimov 
>>> wrote:
>>> 
 Dear community,
 
 I want to share the KIP-123 [1] which is based on issue KAFKA-4144 [2].
>> You
 can check the PR in [3].
 
 I would like to get your comments.
 
 [1]
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
 [2] https://issues.apache.org/jira/browse/KAFKA-4144
 [3] https://github.com/apache/kafka/pull/2466
 
 
 Cheers,
 Jeyhun
 --
 -Cheers
 
 Jeyhun
 
>>> 
>> 
>> --
> -Cheers
> 
> Jeyhun

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Eno Thereska]

Makes sense, thank you Dong.

Eno
> On 28 Feb 2017, at 01:51, Dong Lin  wrote:
> 
> Hi Jun,
> 
> In addition to the Eno's reference of why rebuild time with RAID-5 is more
> expensive, another concern is that RAID-5 will fail if more than one disk
> fails. JBOD is still works with 1+ disk failure and has better performance
> with one disk failure. These seems like good argument for using JBOD
> instead of RAID-5.
> 
> If a leader replica goes offline, the broker should first take all actions
> (i.e. remove the partition from fetcher thread) as if it has received
> StopReplicaRequest for this partition because the replica can no longer
> work anyway. It will also respond with error to any ProduceRequest and
> FetchRequest for partition. The broker notifies controller by writing
> notification znode in ZK. The controller learns the disk failure event from
> ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to learn
> that the replica is offline. The controller will then elect new leader for
> this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
> relevant brokers. The broker should stop adjusting the ISR for this
> partition as if the broker is already offline. I am not sure there is any
> inconsistency in broker's behavior when it is leader or follower. Is there
> any concern with this approach?
> 
> Thanks for catching this. I have removed that reference from the KIP.
> 
> Hi Eno,
> 
> Thank you for providing the reference of the RAID-5. In LinkedIn we have 10
> disks per Kafka machine. It will not be a show-stopper operationally for
> LinkedIn if we have to deploy one-broker-per-disk. On the other hand we
> previously discussed the advantage of JBOD vs. one-broker-per-disk or
> one-broker-per-machine. One-broker-per-disk suffers from the problems
> described in the KIP and one-broker-per-machine increases the failure
> caused by disk failure by 10X. Since JBOD is strictly better than either of
> the two, it is also better then one-broker-per-multiple-disk which is
> somewhere between one-broker-per-disk and one-broker-per-machine.
> 
> I personally think the benefits of JBOD design is worth the implementation
> complexity it introduces. I would also argue that it is reasonable for
> Kafka to manage this low level detail because Kafka is already exposing and
> managing replication factor of its data. But whether the complexity is
> worthwhile can be subjective and I can not prove my opinion. I am
> contributing significant amount of time to do this KIP because Kafka
> develops at LinkedIn believes it is useful and worth the effort. Yeah, it
> will be useful to see what everyone else think about it.
> 
> 
> Thanks,
> Dong
> 
> 
> On Mon, Feb 27, 2017 at 1:16 PM, Jun Rao  wrote:
> 
>> Hi, Dong,
>> 
>> For RAID5, I am not sure the rebuild cost is a big concern. If a disk
>> fails, typically an admin has to bring down the broker, replace the failed
>> disk with a new one, trigger the RAID rebuild, and bring up the broker.
>> This way, there is no performance impact at runtime due to rebuild. The
>> benefit is that a broker doesn't fail in a hard way when there is a disk
>> failure and can be brought down in a controlled way for maintenance. While
>> the broker is running with a failed disk, reads may be more expensive since
>> they have to be computed from the parity. However, if most reads are from
>> page cache, this may not be a big issue either. So, it would be useful to
>> do some tests on RAID5 before we completely rule it out.
>> 
>> Regarding whether to remove an offline replica from the fetcher thread
>> immediately. What do we do when a failed replica is a leader? Do we do
>> nothing or mark the replica as not the leader immediately? Intuitively, it
>> seems it's better if the broker acts consistently on a failed replica
>> whether it's a leader or a follower. For ISR churns, I was just pointing
>> out that if we don't send StopReplicaRequest to a broker to be shut down in
>> a controlled way, then the leader will shrink ISR, expand it and shrink it
>> again after the timeout.
>> 
>> The KIP seems to still reference "
>> /broker/topics/[topic]/partitions/[partitionId]/controller_managed_state".
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Sat, Feb 25, 2017 at 7:49 PM, Dong Lin  wrote:
>> 
>>> Hey Jun,
>>> 
>>> Thanks for the suggestion. I think it is a good idea to know put created
>>> flag in ZK and simply specify isNewReplica=true in LeaderAndIsrRequest if
>>> repilcas was in NewReplica state. It will only fail the replica creation
>> in
>>> the scenario that the controller fails after
>>> topic-creation/partition-reassignment/partition-number-change but before
>>> actually sends out the LeaderAndIsrRequest while there is ongoing disk
>>> failure, which should be pretty rare and acceptable. This should simplify
>>> the design of this KIP.
>>> 
>>> Regarding RAID-5, I think the concern with RAID-5/6 is not just about
>>> performance when there is no failure. For example, RAID-5

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

[Jeyhun Karimov]

Hi Eno,

Thanks for feedback. I think you mean [1]. In this KIP we do not consider
the situations you mentioned. So, either we can extend the KIP and solve
mentioned issues  or submit 2 PRs incrementally.

[1] https://issues.apache.org/jira/browse/KAFKA-4785


Cheers,
Jeyhun

On Tue, Feb 28, 2017 at 10:41 AM Eno Thereska 
wrote:

> Hi Jeyhun,
>
> Thanks for the KIP, sorry I'm coming a bit late to the discussion.
>
> One thing I'd like to understand is whether we can avoid situations where
> the user is mixing different times (event time vs. wallclock time) in their
> processing inadvertently. Before this KIP, all the relevant topics have one
> time stamp extractor so that issue does not come up.
>
> What will be the behavior if times mismatch, e.g., for joins?
>
> Thanks
> Eno
>
> > On 22 Feb 2017, at 09:21, Jeyhun Karimov  wrote:
> >
> > Dear community,
> >
> > I would like to get further feedbacks on this KIP (if any).
> >
> > Cheers
> > Jeyhun
> >
> > On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax 
> > wrote:
> >
> >> Mathieu,
> >>
> >> I personally agree with your observation, and we have plans to submit a
> >> KIP like this. If you want to drive this discussion feel free to start
> >> the KIP by yourself!
> >>
> >> Having said that, for this KIP we might want to focus the discussion the
> >> the actual feature that gets added: allowing to specify different
> >> TS-Extractor for different inputs.
> >>
> >>
> >>
> >> -Matthias
> >>
> >> On 2/14/17 4:54 PM, Mathieu Fenniak wrote:
> >>> Hi Jeyhun,
> >>>
> >>> This KIP might not be the appropriate time, but my first thought
> reading
> >> it
> >>> is that it might make sense to introduce a builder-style API rather
> than
> >>> adding a mix of new method overloads with independent optional
> >> parameters.
> >>> :-)
> >>>
> >>> eg. stream(), table(), globalTable(), addSource(), could all accept a
> >>> "TopicReference" parameter that can be built like:
> >>>
> >>
> TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build().
> >>>
> >>> Mathieu
> >>>
> >>>
> >>> On Tue, Feb 14, 2017 at 5:31 PM, Jeyhun Karimov 
> >>> wrote:
> >>>
>  Dear community,
> 
>  I want to share the KIP-123 [1] which is based on issue KAFKA-4144
> [2].
> >> You
>  can check the PR in [3].
> 
>  I would like to get your comments.
> 
>  [1]
> 
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
>  [2] https://issues.apache.org/jira/browse/KAFKA-4144
>  [3] https://github.com/apache/kafka/pull/2466
> 
> 
>  Cheers,
>  Jeyhun
>  --
>  -Cheers
> 
>  Jeyhun
> 
> >>>
> >>
> >> --
> > -Cheers
> >
> > Jeyhun
>
> --
-Cheers

Jeyhun

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Ismael Juma]

Hi Vahid,

Sure, I've added KIP-54. I thought I had done it, sorry for the oversight.

Ismael

On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> +1 on 0.11.0.0.
>
> Can we also include KIP-54 to the list?
> The PR for this KIP is ready for review.
>
> Thanks.
> --Vahid
>
>
>
>
>
> From:   Ismael Juma 
> To: dev@kafka.apache.org
> Date:   02/27/2017 07:47 PM
> Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> Sent by:isma...@gmail.com
>
>
>
> Hi all,
>
> With 0.10.2.0 out of the way, I would like to volunteer to be the release
> manager for our next time-based release. See https://cwiki.apache.org/c
> onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> communication on time-based releases or need a reminder.
>
> I put together a draft release plan with June 2017 as the release month
> (as
> previously agreed) and a list of KIPs that have already been voted:
>
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68716876
>  >*
>
> I haven't set exact dates for the various stages (feature freeze, code
> freeze, etc.) for now as Ewen is going to send out an email with some
> suggested tweaks based on his experience as release manager for 0.10.2.0.
> We can set the exact dates after that discussion.
>
> As we are starting the process early this time, we should expect the
> number
> of KIPs in the plan to grow (so don't worry if your KIP is not there yet),
> but it's good to see that we already have 10 (including 2 merged and 2
> with
> PR reviews in progress).
>
> Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and KIP-101
> (Leader Generation in Replication) require message format changes, which
> typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> it makes sense to also include KIP-106 (Unclean leader election should be
> false by default) and KIP-118 (Drop support for Java 7). We would also
> take
> the chance to remove deprecated code, in that case.
>
> Given the above, how do people feel about 0.11.0.0 as the next Kafka
> version? Please share your thoughts.
>
> Thanks,
> Ismael
>
>
>
>
>

[jira] [Created] (KAFKA-4812) We are facing the same issue as SAMZA-590

[Manjeer Srujan. Y (JIRA)]

Manjeer Srujan. Y created KAFKA-4812:


 Summary: We are facing the same issue as SAMZA-590
 Key: KAFKA-4812
 URL: https://issues.apache.org/jira/browse/KAFKA-4812
 Project: Kafka
  Issue Type: Bug
Reporter: Manjeer Srujan. Y
Priority: Critical


Dead Kafka broker ignores new leader.

We are facing the same issue as samza issue below. But, we couldn't find any 
fix for this in kafka. Pasted the log below for reference.

The kafka client that we are using is below.

group: 'org.apache.kafka', name: 'kafka_2.10', version: '0.8.2.1'

https://issues.apache.org/jira/browse/SAMZA-590

2017-02-28 09:50:53.189 29708 [Thread-11-vendor-index-spout-executor[35 35]] 
ERROR org.apache.storm.daemon.executor -  - java.lang.RuntimeException: 
java.nio.channels.ClosedChannelException
at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:103)
at 
org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129)
at 
org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:648)
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:78)
at 
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
at 
kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127)
at 
kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75)
at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65)
at 
org.apache.storm.kafka.PartitionManager.(PartitionManager.java:94)
at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
... 6 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

[Eno Thereska]

Hi Jeyhun,

I mean something slightly different. In your motivation you say "joining 
multiple streams/tables that require different timestamp extraction methods". I 
wan to understand the scope of this. Is it allowed to have a stream that uses 
wallclock time join a stream that uses event time? (It would be good to give 
some examples in the motivation about scenarios you envision). If the join is 
not allowed, how do you prevent that join from happening? Do you throw an 
exception?

Thanks
Eno


> On 28 Feb 2017, at 10:04, Jeyhun Karimov  wrote:
> 
> Hi Eno,
> 
> Thanks for feedback. I think you mean [1]. In this KIP we do not consider
> the situations you mentioned. So, either we can extend the KIP and solve
> mentioned issues  or submit 2 PRs incrementally.
> 
> [1] https://issues.apache.org/jira/browse/KAFKA-4785
> 
> 
> Cheers,
> Jeyhun
> 
> On Tue, Feb 28, 2017 at 10:41 AM Eno Thereska 
> wrote:
> 
>> Hi Jeyhun,
>> 
>> Thanks for the KIP, sorry I'm coming a bit late to the discussion.
>> 
>> One thing I'd like to understand is whether we can avoid situations where
>> the user is mixing different times (event time vs. wallclock time) in their
>> processing inadvertently. Before this KIP, all the relevant topics have one
>> time stamp extractor so that issue does not come up.
>> 
>> What will be the behavior if times mismatch, e.g., for joins?
>> 
>> Thanks
>> Eno
>> 
>>> On 22 Feb 2017, at 09:21, Jeyhun Karimov  wrote:
>>> 
>>> Dear community,
>>> 
>>> I would like to get further feedbacks on this KIP (if any).
>>> 
>>> Cheers
>>> Jeyhun
>>> 
>>> On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax 
>>> wrote:
>>> 
 Mathieu,
 
 I personally agree with your observation, and we have plans to submit a
 KIP like this. If you want to drive this discussion feel free to start
 the KIP by yourself!
 
 Having said that, for this KIP we might want to focus the discussion the
 the actual feature that gets added: allowing to specify different
 TS-Extractor for different inputs.
 
 
 
 -Matthias
 
 On 2/14/17 4:54 PM, Mathieu Fenniak wrote:
> Hi Jeyhun,
> 
> This KIP might not be the appropriate time, but my first thought
>> reading
 it
> is that it might make sense to introduce a builder-style API rather
>> than
> adding a mix of new method overloads with independent optional
 parameters.
> :-)
> 
> eg. stream(), table(), globalTable(), addSource(), could all accept a
> "TopicReference" parameter that can be built like:
> 
 
>> TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build().
> 
> Mathieu
> 
> 
> On Tue, Feb 14, 2017 at 5:31 PM, Jeyhun Karimov 
> wrote:
> 
>> Dear community,
>> 
>> I want to share the KIP-123 [1] which is based on issue KAFKA-4144
>> [2].
 You
>> can check the PR in [3].
>> 
>> I would like to get your comments.
>> 
>> [1]
>> 
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
>> [2] https://issues.apache.org/jira/browse/KAFKA-4144
>> [3] https://github.com/apache/kafka/pull/2466
>> 
>> 
>> Cheers,
>> Jeyhun
>> --
>> -Cheers
>> 
>> Jeyhun
>> 
> 
 
 --
>>> -Cheers
>>> 
>>> Jeyhun
>> 
>> --
> -Cheers
> 
> Jeyhun

[jira] [Updated] (KAFKA-4812) We are facing the same issue as SAMZA-590 for kafka

[Manjeer Srujan. Y (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manjeer Srujan. Y updated KAFKA-4812:
-
Summary: We are facing the same issue as SAMZA-590 for kafka  (was: We are 
facing the same issue as SAMZA-590)

> We are facing the same issue as SAMZA-590 for kafka
> ---
>
> Key: KAFKA-4812
> URL: https://issues.apache.org/jira/browse/KAFKA-4812
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manjeer Srujan. Y
>Priority: Critical
>
> Dead Kafka broker ignores new leader.
> We are facing the same issue as samza issue below. But, we couldn't find any 
> fix for this in kafka. Pasted the log below for reference.
> The kafka client that we are using is below.
> group: 'org.apache.kafka', name: 'kafka_2.10', version: '0.8.2.1'
> https://issues.apache.org/jira/browse/SAMZA-590
> 2017-02-28 09:50:53.189 29708 [Thread-11-vendor-index-spout-executor[35 35]] 
> ERROR org.apache.storm.daemon.executor -  - java.lang.RuntimeException: 
> java.nio.channels.ClosedChannelException
> at 
> org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:103)
> at 
> org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
> at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129)
> at 
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:648)
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> at clojure.lang.AFn.run(AFn.java:22)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
> at 
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:78)
> at 
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
> at 
> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127)
> at 
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
> at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75)
> at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65)
> at 
> org.apache.storm.kafka.PartitionManager.(PartitionManager.java:94)
> at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
> ... 6 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Ismael Juma]

Hi everyone,

Since the few who responded in the discuss thread were in favour and there
were no objections, I would like to initiate the voting process for
KIP-119: Drop Support for Scala 2.10 in Kafka 0.11:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11

The vote will run for a minimum of 72 hours.

Thanks,
Ismael

[jira] [Commented] (KAFKA-4812) We are facing the same issue as SAMZA-590 for kafka

[Manjeer Srujan. Y (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887779#comment-15887779
 ] 

Manjeer Srujan. Y commented on KAFKA-4812:
--

[~criccomini]: Seems like you have fixed it in Samza as per the SAMZA-590. Can 
you please let me know if the same issue is fixed for kafka also ? 

> We are facing the same issue as SAMZA-590 for kafka
> ---
>
> Key: KAFKA-4812
> URL: https://issues.apache.org/jira/browse/KAFKA-4812
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manjeer Srujan. Y
>Priority: Critical
>
> Dead Kafka broker ignores new leader.
> We are facing the same issue as samza issue below. But, we couldn't find any 
> fix for this in kafka. Pasted the log below for reference.
> The kafka client that we are using is below.
> group: 'org.apache.kafka', name: 'kafka_2.10', version: '0.8.2.1'
> https://issues.apache.org/jira/browse/SAMZA-590
> 2017-02-28 09:50:53.189 29708 [Thread-11-vendor-index-spout-executor[35 35]] 
> ERROR org.apache.storm.daemon.executor -  - java.lang.RuntimeException: 
> java.nio.channels.ClosedChannelException
> at 
> org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:103)
> at 
> org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
> at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129)
> at 
> org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:648)
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> at clojure.lang.AFn.run(AFn.java:22)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
> at 
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:78)
> at 
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
> at 
> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127)
> at 
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
> at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75)
> at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65)
> at 
> org.apache.storm.kafka.PartitionManager.(PartitionManager.java:94)
> at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
> ... 6 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Ofir Manor]

This is definitely a major release, looks already quite exciting...
I don't want to start a bike-shading argument, but a few people have told
me in the past year that once exactly-once delivery lands, Kafka would
likely bump to 1.0. I do like it, but don't feel strongly either way.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma  wrote:

> Hi Vahid,
>
> Sure, I've added KIP-54. I thought I had done it, sorry for the oversight.
>
> Ismael
>
> On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > +1 on 0.11.0.0.
> >
> > Can we also include KIP-54 to the list?
> > The PR for this KIP is ready for review.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> >
> > From:   Ismael Juma 
> > To: dev@kafka.apache.org
> > Date:   02/27/2017 07:47 PM
> > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > Sent by:isma...@gmail.com
> >
> >
> >
> > Hi all,
> >
> > With 0.10.2.0 out of the way, I would like to volunteer to be the release
> > manager for our next time-based release. See https://cwiki.apache.org/c
> > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > communication on time-based releases or need a reminder.
> >
> > I put together a draft release plan with June 2017 as the release month
> > (as
> > previously agreed) and a list of KIPs that have already been voted:
> >
> > *https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68716876
> >  action?pageId=68716876
> > >*
> >
> > I haven't set exact dates for the various stages (feature freeze, code
> > freeze, etc.) for now as Ewen is going to send out an email with some
> > suggested tweaks based on his experience as release manager for 0.10.2.0.
> > We can set the exact dates after that discussion.
> >
> > As we are starting the process early this time, we should expect the
> > number
> > of KIPs in the plan to grow (so don't worry if your KIP is not there
> yet),
> > but it's good to see that we already have 10 (including 2 merged and 2
> > with
> > PR reviews in progress).
> >
> > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> KIP-101
> > (Leader Generation in Replication) require message format changes, which
> > typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> > it makes sense to also include KIP-106 (Unclean leader election should be
> > false by default) and KIP-118 (Drop support for Java 7). We would also
> > take
> > the chance to remove deprecated code, in that case.
> >
> > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > version? Please share your thoughts.
> >
> > Thanks,
> > Ismael
> >
> >
> >
> >
> >
>

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Ismael Juma]

Hi Ofir,

Thanks for the feedback. This will be the first release with exactly-once.
I think we should give ourselves at least one stabilisation/polish/feedback
cycle before we go for 1.0. :)

Ismael

On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor  wrote:

> This is definitely a major release, looks already quite exciting...
> I don't want to start a bike-shading argument, but a few people have told
> me in the past year that once exactly-once delivery lands, Kafka would
> likely bump to 1.0. I do like it, but don't feel strongly either way.
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma  wrote:
>
> > Hi Vahid,
> >
> > Sure, I've added KIP-54. I thought I had done it, sorry for the
> oversight.
> >
> > Ismael
> >
> > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > +1 on 0.11.0.0.
> > >
> > > Can we also include KIP-54 to the list?
> > > The PR for this KIP is ready for review.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > >
> > > From:   Ismael Juma 
> > > To: dev@kafka.apache.org
> > > Date:   02/27/2017 07:47 PM
> > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > Sent by:isma...@gmail.com
> > >
> > >
> > >
> > > Hi all,
> > >
> > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> release
> > > manager for our next time-based release. See
> https://cwiki.apache.org/c
> > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > > communication on time-based releases or need a reminder.
> > >
> > > I put together a draft release plan with June 2017 as the release month
> > > (as
> > > previously agreed) and a list of KIPs that have already been voted:
> > >
> > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=68716876
> > >  > action?pageId=68716876
> > > >*
> > >
> > > I haven't set exact dates for the various stages (feature freeze, code
> > > freeze, etc.) for now as Ewen is going to send out an email with some
> > > suggested tweaks based on his experience as release manager for
> 0.10.2.0.
> > > We can set the exact dates after that discussion.
> > >
> > > As we are starting the process early this time, we should expect the
> > > number
> > > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > yet),
> > > but it's good to see that we already have 10 (including 2 merged and 2
> > > with
> > > PR reviews in progress).
> > >
> > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > KIP-101
> > > (Leader Generation in Replication) require message format changes,
> which
> > > typically imply a major version bump (i.e. 0.11.0.0). If we do that,
> then
> > > it makes sense to also include KIP-106 (Unclean leader election should
> be
> > > false by default) and KIP-118 (Drop support for Java 7). We would also
> > > take
> > > the chance to remove deprecated code, in that case.
> > >
> > > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > > version? Please share your thoughts.
> > >
> > > Thanks,
> > > Ismael
> > >
> > >
> > >
> > >
> > >
> >
>

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Ismael Juma]

Hi Jeff,

I share your desire to streamline the experience and remove options that
don't make sense to new users. I should, however, point out that we won't
be able to remove the old consumers in 0.11.0.0 as they haven't been
deprecated yet. Hopefully that will happen in 0.12.0.0/1.0.0.

Ismael

On Tue, Feb 28, 2017 at 4:24 AM, Jeff Widman  wrote:

> +1 for major version bump.
>
> A good bit of deprecated code I would like to see removed especially on old
> consumer side plus a few other settings defaults changed such as the brief
> discussion on mirrormaker options a few months back. Just be good to
> continue to make the new user experience a lot more streamlined so they're
> not wondering about all these variations on consumers, CLI scripts etc.
>
> On Feb 27, 2017 8:14 PM, "Becket Qin"  wrote:
>
> > Hi Ismael,
> >
> > Thanks for volunteering on the new release.
> >
> > I think 0.11.0.0 makes a lot of sense given the new big features we are
> > intended to include.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Feb 27, 2017 at 7:47 PM, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> release
> > > manager for our next time-based release. See
> https://cwiki.apache.org/c
> > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > > communication on time-based releases or need a reminder.
> > >
> > > I put together a draft release plan with June 2017 as the release month
> > (as
> > > previously agreed) and a list of KIPs that have already been voted:
> > >
> > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=68716876
> > >  > action?pageId=68716876
> > > >*
> > >
> > > I haven't set exact dates for the various stages (feature freeze, code
> > > freeze, etc.) for now as Ewen is going to send out an email with some
> > > suggested tweaks based on his experience as release manager for
> 0.10.2.0.
> > > We can set the exact dates after that discussion.
> > >
> > > As we are starting the process early this time, we should expect the
> > number
> > > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > yet),
> > > but it's good to see that we already have 10 (including 2 merged and 2
> > with
> > > PR reviews in progress).
> > >
> > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > KIP-101
> > > (Leader Generation in Replication) require message format changes,
> which
> > > typically imply a major version bump (i.e. 0.11.0.0). If we do that,
> then
> > > it makes sense to also include KIP-106 (Unclean leader election should
> be
> > > false by default) and KIP-118 (Drop support for Java 7). We would also
> > take
> > > the chance to remove deprecated code, in that case.
> > >
> > > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > > version? Please share your thoughts.
> > >
> > > Thanks,
> > > Ismael
> > >
> >
>

Re: [VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Dongjin Lee]

 
 
+1.
 

 
Best,
 
Dongjin
   
 
 --
 
 
 
 
Dongjin Lee
 
 
 

   
Software developer in Line+.
 
So interested in massive-scale machine learning.
 
 

 facebook:   www.facebook.com/dongjin.lee.kr 
(http://www.facebook.com/dongjin.lee.kr)  
 
linkedin:   kr.linkedin.com/in/dongjinleekr 
(http://kr.linkedin.com/in/dongjinleekr)
 
 
github:   (http://goog_969573159/)github.com/dongjinleekr 
(http://github.com/dongjinleekr)
 
twitter:   www.twitter.com/dongjinleekr (http://www.twitter.com/dongjinleekr)
 
 
 
 
 
 
 
 

 
 
>  
> On Feb 28, 2017 at 6:40 PM,  mailto:ism...@juma.me.uk)>  wrote:
>  
>  
>  
>  Hi everyone, 
>
> Since the few who responded in the discuss thread were in favour and there 
> were no objections, I would like to initiate the voting process for 
> KIP-119: Drop Support for Scala 2.10 in Kafka 0.11: 
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
>  
>
> The vote will run for a minimum of 72 hours. 
>
> Thanks, 
> Ismael 
>  

Re: [VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Molnár Bálint]

+1

2017-02-28 12:17 GMT+01:00 Dongjin Lee :

>
>
> +1.
>
>
>
> Best,
>
> Dongjin
>
>
>  --
>
>
>
>
> Dongjin Lee
>
>
>
>
>
> Software developer in Line+.
>
> So interested in massive-scale machine learning.
>
>
>
>  facebook:   www.facebook.com/dongjin.lee.kr (http://www.facebook.com/
> dongjin.lee.kr)
>
> linkedin:   kr.linkedin.com/in/dongjinleekr (http://kr.linkedin.com/in/
> dongjinleekr)
>
>
> github:   (http://goog_969573159/)github.com/dongjinleekr (
> http://github.com/dongjinleekr)
>
> twitter:   www.twitter.com/dongjinleekr (http://www.twitter.com/
> dongjinleekr)
>
>
>
>
>
>
>
>
>
>
>
> >
> > On Feb 28, 2017 at 6:40 PM,  mailto:ism...@juma.me.uk)>
> wrote:
> >
> >
> >
> >  Hi everyone,
> >
> > Since the few who responded in the discuss thread were in favour and
> there
> > were no objections, I would like to initiate the voting process for
> > KIP-119: Drop Support for Scala 2.10 in Kafka 0.11:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> > Ismael
> >
>

[GitHub] kafka pull request #2607: MINOR: Fix typo in javadoc of `flatMapValues`

[miguno]

GitHub user miguno opened a pull request:

https://github.com/apache/kafka/pull/2607

MINOR: Fix typo in javadoc of `flatMapValues`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miguno/kafka trunk-flatMapValues-docstring

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2607.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2607


commit 5ce051b0b437eb2e30413f97d2e9e03959aa7532
Author: Michael G. Noll 
Date:   2017-02-28T11:47:09Z

Fix typo in javadoc of `flatMapValues`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-82 - Add Record Headers

[Ismael Juma]

Hi Becket,

Comments inline.

On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin  wrote:
>
> 1. Regarding the mutability.
>
> I think it would be a big convenience to have headers mutable during
> certain stage in the message life cycle for the use cases you mentioned. I
> agree there is a material benefit especially given that we may have to
> modify the headers for each message.
>
> That said, I also think it is fair to say that in the producer, in order to
> guarantee the correctness of the entire logic, it is necessary that at some
> point we need to make producer record immutable. For example we probably
> don't want to see that users accidentally updated the headers when the
> producer is doing the serialization or compression.
>
> Given that, would it be possible to make Headers to be able to switch from
> mutable to immutable? We have done this for the Batch in the producer. For
> example, initially the headers are mutable, but after it has gone through
> all the interceptors, we can call Headers.close() to make it immutable
> afterwards.
>

The difference is that the batch is an internal class that is not exposed
to users. Can you please explain what happens if a user tries to send the
same ProducerRecord twice? Would an interceptor fail when trying to mutate
the header that is now closed? Or did you have something else in mind?

Thanks,
Ismael

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Rajini Sivaram]

Ismael,

Thank you for volunteering to manage the next release.

+1 for 0.11.0.0

Regards,

Rajini

On Tue, Feb 28, 2017 at 11:02 AM, Ismael Juma  wrote:

> Hi Ofir,
>
> Thanks for the feedback. This will be the first release with exactly-once.
> I think we should give ourselves at least one stabilisation/polish/feedback
> cycle before we go for 1.0. :)
>
> Ismael
>
> On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor 
> wrote:
>
> > This is definitely a major release, looks already quite exciting...
> > I don't want to start a bike-shading argument, but a few people have told
> > me in the past year that once exactly-once delivery lands, Kafka would
> > likely bump to 1.0. I do like it, but don't feel strongly either way.
> >
> > Ofir Manor
> >
> > Co-Founder & CTO | Equalum
> >
> > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> >
> > On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma  wrote:
> >
> > > Hi Vahid,
> > >
> > > Sure, I've added KIP-54. I thought I had done it, sorry for the
> > oversight.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > > > +1 on 0.11.0.0.
> > > >
> > > > Can we also include KIP-54 to the list?
> > > > The PR for this KIP is ready for review.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From:   Ismael Juma 
> > > > To: dev@kafka.apache.org
> > > > Date:   02/27/2017 07:47 PM
> > > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > > Sent by:isma...@gmail.com
> > > >
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> > release
> > > > manager for our next time-based release. See
> > https://cwiki.apache.org/c
> > > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed
> previous
> > > > communication on time-based releases or need a reminder.
> > > >
> > > > I put together a draft release plan with June 2017 as the release
> month
> > > > (as
> > > > previously agreed) and a list of KIPs that have already been voted:
> > > >
> > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68716876
> > > >  > > action?pageId=68716876
> > > > >*
> > > >
> > > > I haven't set exact dates for the various stages (feature freeze,
> code
> > > > freeze, etc.) for now as Ewen is going to send out an email with some
> > > > suggested tweaks based on his experience as release manager for
> > 0.10.2.0.
> > > > We can set the exact dates after that discussion.
> > > >
> > > > As we are starting the process early this time, we should expect the
> > > > number
> > > > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > > yet),
> > > > but it's good to see that we already have 10 (including 2 merged and
> 2
> > > > with
> > > > PR reviews in progress).
> > > >
> > > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > > KIP-101
> > > > (Leader Generation in Replication) require message format changes,
> > which
> > > > typically imply a major version bump (i.e. 0.11.0.0). If we do that,
> > then
> > > > it makes sense to also include KIP-106 (Unclean leader election
> should
> > be
> > > > false by default) and KIP-118 (Drop support for Java 7). We would
> also
> > > > take
> > > > the chance to remove deprecated code, in that case.
> > > >
> > > > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > > > version? Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Tom Crayford]

+1 (non-binding)

On Tue, 28 Feb 2017 at 11:42, Molnár Bálint  wrote:

> +1
>
> 2017-02-28 12:17 GMT+01:00 Dongjin Lee :
>
> >
> >
> > +1.
> >
> >
> >
> > Best,
> >
> > Dongjin
> >
> >
> >  --
> >
> >
> >
> >
> > Dongjin Lee
> >
> >
> >
> >
> >
> > Software developer in Line+.
> >
> > So interested in massive-scale machine learning.
> >
> >
> >
> >  facebook:   www.facebook.com/dongjin.lee.kr (http://www.facebook.com/
> > dongjin.lee.kr)
> >
> > linkedin:   kr.linkedin.com/in/dongjinleekr (http://kr.linkedin.com/in/
> > dongjinleekr)
> >
> >
> > github:   (http://goog_969573159/)github.com/dongjinleekr (
> > http://github.com/dongjinleekr)
> >
> > twitter:   www.twitter.com/dongjinleekr (http://www.twitter.com/
> > dongjinleekr)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > >
> > > On Feb 28, 2017 at 6:40 PM,  mailto:ism...@juma.me.uk)>
> > wrote:
> > >
> > >
> > >
> > >  Hi everyone,
> > >
> > > Since the few who responded in the discuss thread were in favour and
> > there
> > > were no objections, I would like to initiate the voting process for
> > > KIP-119: Drop Support for Scala 2.10 in Kafka 0.11:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
> > >
> > > The vote will run for a minimum of 72 hours.
> > >
> > > Thanks,
> > > Ismael
> > >
> >
>

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[Rajini Sivaram (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887913#comment-15887913
 ] 

Rajini Sivaram commented on KAFKA-4779:
---

The new failure looks like a very different issue. The consumer logs show:

{quote}
[2017-02-26 05:14:28,833] WARN Error while fetching metadata with correlation 
id 10712 : test_topic=LEADER_NOT_AVAILABLE 
(org.apache.kafka.clients.NetworkClient)

[2017-02-26 05:14:29,655] DEBUG Received successful JoinGroup response for 
group group: org.apache.kafka.common.requests.JoinGroupResponse@7802468d 
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2017-02-26 05:14:29,655] DEBUG Performing assignment for group group using 
strategy range with subscriptions 
console-consumer-810cd1cc-eb5b-4d6d-bfb5-6e094e107d14=Subscription(topics=[test_topic])(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-02-26 05:14:29,655] DEBUG Skipping assignment for topic test_topic since 
no metadata is available 
(org.apache.kafka.clients.consumer.internals.AbstractPartitionAssignor)
[2017-02-26 05:14:29,655] WARN The following subscribed topics are not assigned 
to any members in the group group : [test_topic]  
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
{quote}

Even though partitions of the subscribed topic could not be assigned because no 
metadata is available due to a transient error resulting from broker restart, 
no metadata refresh is requested. The consumer times out after a minute. 
Metadata expiry is 5 minutes, so rebalancing will be delayed for 5 minutes. 
Consumers currently request metadata when leader is not known during fetching, 
but not if assignment cannot be performed. I think we should request metadata 
sooner for this case. Will submit a PR.


> Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py
> 
>
> Key: KAFKA-4779
> URL: https://issues.apache.org/jira/browse/KAFKA-4779
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Rajini Sivaram
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> This test failed on 01/29, on both trunk and 0.10.2, error message:
> {noformat}
> The consumer has terminated, or timed out, on node ubuntu@worker3.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py",
>  line 148, in test_rolling_upgrade_phase_two
> self.run_produce_consume_validate(self.roll_in_secured_settings, 
> client_protocol, broker_protocol)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 100, in run_produce_consume_validate
> self.stop_producer_and_consumer()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 87, in stop_producer_and_consumer
> self.check_alive()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in check_alive
> raise Exception(msg)
> Exception: The consumer has terminated, or timed out, on node ubuntu@worker3.
> {noformat}
> Looks like the console consumer times out: 
> {noformat}
> [2017-01-30 04:56:00,972] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> kafka.consumer.ConsumerTimeoutException
> at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:90)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {noformat}
> A bunch of these security_rolling_upgrade tests failed, and in all cases, the 
> producer produced ~15k messages, of which ~7k were acked, and the consumer 
> only got around ~2600 before timing out

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887934#comment-15887934
 ] 

Ismael Juma commented on KAFKA-4779:


Thanks [~rsivaram]! Hopefully this is the last issue uncovered by this system 
test. :)

> Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py
> 
>
> Key: KAFKA-4779
> URL: https://issues.apache.org/jira/browse/KAFKA-4779
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Rajini Sivaram
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> This test failed on 01/29, on both trunk and 0.10.2, error message:
> {noformat}
> The consumer has terminated, or timed out, on node ubuntu@worker3.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py",
>  line 148, in test_rolling_upgrade_phase_two
> self.run_produce_consume_validate(self.roll_in_secured_settings, 
> client_protocol, broker_protocol)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 100, in run_produce_consume_validate
> self.stop_producer_and_consumer()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 87, in stop_producer_and_consumer
> self.check_alive()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in check_alive
> raise Exception(msg)
> Exception: The consumer has terminated, or timed out, on node ubuntu@worker3.
> {noformat}
> Looks like the console consumer times out: 
> {noformat}
> [2017-01-30 04:56:00,972] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> kafka.consumer.ConsumerTimeoutException
> at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:90)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {noformat}
> A bunch of these security_rolling_upgrade tests failed, and in all cases, the 
> producer produced ~15k messages, of which ~7k were acked, and the consumer 
> only got around ~2600 before timing out. 
> There are a lot of messages like the following for different request types on 
> the producer and consumer:
> {noformat}
> [2017-01-30 05:13:35,954] WARN Received unknown topic or partition error in 
> produce request on partition test_topic-0. The topic/partition may not exist 
> or the user may not have Describe access to it 
> (org.apache.kafka.clients.producer.internals.Sender)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Michael Noll]

Thanks, Jeyhun!

I observed that there is still on ongoing conversation in the DISCUSS
thread, so perhaps this voting thread was started a bit too early?

-Michael




On Tue, Feb 28, 2017 at 10:35 AM, Damian Guy  wrote:

> Thanks for the KIP Jeyhun!
>
> +1
>
> On Tue, 28 Feb 2017 at 08:59 Jeyhun Karimov  wrote:
>
> > Dear community,
> >
> > I'd like to start the vote for KIP-123:
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68714788
> >
> >
> > Cheers,
> > Jeyhun
> > --
> > -Cheers
> >
> > Jeyhun
> >
>

[jira] [Commented] (KAFKA-3990) Kafka New Producer may raise an OutOfMemoryError

[Adrian McCague (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887957#comment-15887957
 ] 

Adrian McCague commented on KAFKA-3990:
---

I have narrowed down my particular issues to being caused by the confluent 
monitoring interceptors so it may not be Kafka specific.

> Kafka New Producer may raise an OutOfMemoryError
> 
>
> Key: KAFKA-3990
> URL: https://issues.apache.org/jira/browse/KAFKA-3990
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.1
> Environment: Docker, Base image : CentOS
> Java 8u77
> Marathon
>Reporter: Brice Dutheil
> Attachments: app-producer-config.log, kafka-broker-logs.zip
>
>
> We are regularly seeing OOME errors on a kafka producer, we first saw :
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
> at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
>  ~[kafka-clients-0.9.0.1.jar:na]
> at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
>  ~[kafka-clients-0.9.0.1.jar:na]
> at 
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at 
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) 
> ~[kafka-clients-0.9.0.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77]
> {code}
> This line refer to a buffer allocation {{ByteBuffer.allocate(receiveSize)}} 
> (see 
> https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L93)
> Usually the app runs fine within 200/400 MB heap and a 64 MB Metaspace. And 
> we are producing small messages 500B at most.
> Also the error don't appear on the devlopment environment, in order to 
> identify the issue we tweaked the code to give us actual data of the 
> allocation size, we got this stack :
> {code}
> 09:55:49.484 [auth] [kafka-producer-network-thread | producer-1] WARN  
> o.a.k.c.n.NetworkReceive HEAP-ISSUE: constructor : Integer='-1', String='-1'
> 09:55:49.485 [auth] [kafka-producer-network-thread | producer-1] WARN  
> o.a.k.c.n.NetworkReceive HEAP-ISSUE: method : 
> NetworkReceive.readFromReadableChannel.receiveSize=1213486160
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to /tmp/tomcat.hprof ...
> Heap dump file created [69583827 bytes in 0.365 secs]
> 09:55:50.324 [auth] [kafka-producer-network-thread | producer-1] ERROR 
> o.a.k.c.utils.KafkaThread Uncaught exception in kafka-producer-network-thread 
> | producer-1: 
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77]
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
>  ~[kafka-clients-0.9.0.1.jar:na]
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
>  ~[kafka-clients-0.9.0.1.jar:na]
>   at 
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at org.apache.kafka.common.network.Selector.poll(Selector.java:286) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) 
> ~[kafka-clients-0.9.0.1.jar:na]
>   at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77]
> {code}
> Notice the size to allocate {{1213486160}} ~1.2 GB. I'm not yet sure how this 
> size is initialised.
> Notice as well that every time this OOME appear the {{NetworkReceive}} 
> constructor at 
> https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L49
>  receive the parameters : {{maxSize=-1}}, {{source="-1"}}
> We 

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Eno Thereska]

That's ok, that was my fault for looking at this thread late. We can leave the 
vote thread as is and simultaneously answer any remaining questions on the 
discuss thread.

Thanks
Eno
> On 28 Feb 2017, at 12:55, Michael Noll  wrote:
> 
> Thanks, Jeyhun!
> 
> I observed that there is still on ongoing conversation in the DISCUSS
> thread, so perhaps this voting thread was started a bit too early?
> 
> -Michael
> 
> 
> 
> 
> On Tue, Feb 28, 2017 at 10:35 AM, Damian Guy  wrote:
> 
>> Thanks for the KIP Jeyhun!
>> 
>> +1
>> 
>> On Tue, 28 Feb 2017 at 08:59 Jeyhun Karimov  wrote:
>> 
>>> Dear community,
>>> 
>>> I'd like to start the vote for KIP-123:
>>> https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=68714788
>>> 
>>> 
>>> Cheers,
>>> Jeyhun
>>> --
>>> -Cheers
>>> 
>>> Jeyhun
>>> 
>> 

[GitHub] kafka pull request #2608: KAFKA-4779: Request metadata in consumer if partit...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2608

KAFKA-4779: Request metadata in consumer if partitions unavailable

If leader node of one more more partitions in a consumer subscription are 
temporarily unavailable, request metadata refresh so that partitions skipped 
for assignment dont have to wait for metadata expiry before reassignment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka 
KAFKA-4779-consumer-metadata

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2608.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2608


commit af5ef2b98314653de5ae098e8c45eb84b81685e7
Author: Rajini Sivaram 
Date:   2017-02-28T12:09:44Z

KAFKA-4779: Request metadata in consumer if partitions unavailable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887978#comment-15887978
 ] 

ASF GitHub Bot commented on KAFKA-4779:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2608

KAFKA-4779: Request metadata in consumer if partitions unavailable

If leader node of one more more partitions in a consumer subscription are 
temporarily unavailable, request metadata refresh so that partitions skipped 
for assignment dont have to wait for metadata expiry before reassignment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka 
KAFKA-4779-consumer-metadata

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2608.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2608


commit af5ef2b98314653de5ae098e8c45eb84b81685e7
Author: Rajini Sivaram 
Date:   2017-02-28T12:09:44Z

KAFKA-4779: Request metadata in consumer if partitions unavailable




> Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py
> 
>
> Key: KAFKA-4779
> URL: https://issues.apache.org/jira/browse/KAFKA-4779
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Rajini Sivaram
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> This test failed on 01/29, on both trunk and 0.10.2, error message:
> {noformat}
> The consumer has terminated, or timed out, on node ubuntu@worker3.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py",
>  line 148, in test_rolling_upgrade_phase_two
> self.run_produce_consume_validate(self.roll_in_secured_settings, 
> client_protocol, broker_protocol)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 100, in run_produce_consume_validate
> self.stop_producer_and_consumer()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 87, in stop_producer_and_consumer
> self.check_alive()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in check_alive
> raise Exception(msg)
> Exception: The consumer has terminated, or timed out, on node ubuntu@worker3.
> {noformat}
> Looks like the console consumer times out: 
> {noformat}
> [2017-01-30 04:56:00,972] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> kafka.consumer.ConsumerTimeoutException
> at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:90)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {noformat}
> A bunch of these security_rolling_upgrade tests failed, and in all cases, the 
> producer produced ~15k messages, of which ~7k were acked, and the consumer 
> only got around ~2600 before timing out. 
> There are a lot of messages like the following for different request types on 
> the producer and consumer:
> {noformat}
> [2017-01-30 05:13:35,954] WARN Received unknown topic or partition error in 
> produce request on partition test_topic-0. The topic/partition may not exist 
> or the user may not have Describe access to it 
> (org.apache.kafka.clients.producer.internals.Sender)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4813) 2h6R1

[Vamsi Jakkula (JIRA)]

Vamsi Jakkula created KAFKA-4813:


 Summary: 2h6R1
 Key: KAFKA-4813
 URL: https://issues.apache.org/jira/browse/KAFKA-4813
 Project: Kafka
  Issue Type: Bug
Reporter: Vamsi Jakkula


Creating of an issue using project keys and issue type names using the REST API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Ofir Manor]

Thanks Ismael, makes perfect sense :)

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Feb 28, 2017 at 1:02 PM, Ismael Juma  wrote:

> Hi Ofir,
>
> Thanks for the feedback. This will be the first release with exactly-once.
> I think we should give ourselves at least one stabilisation/polish/feedback
> cycle before we go for 1.0. :)
>
> Ismael
>
> On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor 
> wrote:
>
> > This is definitely a major release, looks already quite exciting...
> > I don't want to start a bike-shading argument, but a few people have told
> > me in the past year that once exactly-once delivery lands, Kafka would
> > likely bump to 1.0. I do like it, but don't feel strongly either way.
> >
> > Ofir Manor
> >
> > Co-Founder & CTO | Equalum
> >
> > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> >
> > On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma  wrote:
> >
> > > Hi Vahid,
> > >
> > > Sure, I've added KIP-54. I thought I had done it, sorry for the
> > oversight.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > > > +1 on 0.11.0.0.
> > > >
> > > > Can we also include KIP-54 to the list?
> > > > The PR for this KIP is ready for review.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From:   Ismael Juma 
> > > > To: dev@kafka.apache.org
> > > > Date:   02/27/2017 07:47 PM
> > > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > > Sent by:isma...@gmail.com
> > > >
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> > release
> > > > manager for our next time-based release. See
> > https://cwiki.apache.org/c
> > > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed
> previous
> > > > communication on time-based releases or need a reminder.
> > > >
> > > > I put together a draft release plan with June 2017 as the release
> month
> > > > (as
> > > > previously agreed) and a list of KIPs that have already been voted:
> > > >
> > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68716876
> > > >  > > action?pageId=68716876
> > > > >*
> > > >
> > > > I haven't set exact dates for the various stages (feature freeze,
> code
> > > > freeze, etc.) for now as Ewen is going to send out an email with some
> > > > suggested tweaks based on his experience as release manager for
> > 0.10.2.0.
> > > > We can set the exact dates after that discussion.
> > > >
> > > > As we are starting the process early this time, we should expect the
> > > > number
> > > > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > > yet),
> > > > but it's good to see that we already have 10 (including 2 merged and
> 2
> > > > with
> > > > PR reviews in progress).
> > > >
> > > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > > KIP-101
> > > > (Leader Generation in Replication) require message format changes,
> > which
> > > > typically imply a major version bump (i.e. 0.11.0.0). If we do that,
> > then
> > > > it makes sense to also include KIP-106 (Unclean leader election
> should
> > be
> > > > false by default) and KIP-118 (Drop support for Java 7). We would
> also
> > > > take
> > > > the chance to remove deprecated code, in that case.
> > > >
> > > > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > > > version? Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Molnár Bálint]

Thanks Ismael, +1 for 0.11.0.0

2017-02-28 15:20 GMT+01:00 Ofir Manor :

> Thanks Ismael, makes perfect sense :)
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Tue, Feb 28, 2017 at 1:02 PM, Ismael Juma  wrote:
>
> > Hi Ofir,
> >
> > Thanks for the feedback. This will be the first release with
> exactly-once.
> > I think we should give ourselves at least one
> stabilisation/polish/feedback
> > cycle before we go for 1.0. :)
> >
> > Ismael
> >
> > On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor 
> > wrote:
> >
> > > This is definitely a major release, looks already quite exciting...
> > > I don't want to start a bike-shading argument, but a few people have
> told
> > > me in the past year that once exactly-once delivery lands, Kafka would
> > > likely bump to 1.0. I do like it, but don't feel strongly either way.
> > >
> > > Ofir Manor
> > >
> > > Co-Founder & CTO | Equalum
> > >
> > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma 
> wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > Sure, I've added KIP-54. I thought I had done it, sorry for the
> > > oversight.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > >
> > > > > +1 on 0.11.0.0.
> > > > >
> > > > > Can we also include KIP-54 to the list?
> > > > > The PR for this KIP is ready for review.
> > > > >
> > > > > Thanks.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From:   Ismael Juma 
> > > > > To: dev@kafka.apache.org
> > > > > Date:   02/27/2017 07:47 PM
> > > > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > > > Sent by:isma...@gmail.com
> > > > >
> > > > >
> > > > >
> > > > > Hi all,
> > > > >
> > > > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> > > release
> > > > > manager for our next time-based release. See
> > > https://cwiki.apache.org/c
> > > > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed
> > previous
> > > > > communication on time-based releases or need a reminder.
> > > > >
> > > > > I put together a draft release plan with June 2017 as the release
> > month
> > > > > (as
> > > > > previously agreed) and a list of KIPs that have already been voted:
> > > > >
> > > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=68716876
> > > > >  > > > action?pageId=68716876
> > > > > >*
> > > > >
> > > > > I haven't set exact dates for the various stages (feature freeze,
> > code
> > > > > freeze, etc.) for now as Ewen is going to send out an email with
> some
> > > > > suggested tweaks based on his experience as release manager for
> > > 0.10.2.0.
> > > > > We can set the exact dates after that discussion.
> > > > >
> > > > > As we are starting the process early this time, we should expect
> the
> > > > > number
> > > > > of KIPs in the plan to grow (so don't worry if your KIP is not
> there
> > > > yet),
> > > > > but it's good to see that we already have 10 (including 2 merged
> and
> > 2
> > > > > with
> > > > > PR reviews in progress).
> > > > >
> > > > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > > > KIP-101
> > > > > (Leader Generation in Replication) require message format changes,
> > > which
> > > > > typically imply a major version bump (i.e. 0.11.0.0). If we do
> that,
> > > then
> > > > > it makes sense to also include KIP-106 (Unclean leader election
> > should
> > > be
> > > > > false by default) and KIP-118 (Drop support for Java 7). We would
> > also
> > > > > take
> > > > > the chance to remove deprecated code, in that case.
> > > > >
> > > > > Given the above, how do people feel about 0.11.0.0 as the next
> Kafka
> > > > > version? Please share your thoughts.
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Commented] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

[Federico Giraud (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888126#comment-15888126
 ] 

Federico Giraud commented on KAFKA-4669:


We had the same issue with a Kafka 0.8.2.0 producer and a Kafka 0.9 cluster. 
The bug seems to have been triggered by a change in the ISR (a broker fell out 
of sync and cause a rebalance on multiple topics). A producer that was sending 
messages to multiple topics in the cluster reporter the following exception 
multiple times, within a few seconds:

{code}
[2017-02-19 06:02:05,172] ERROR {kafka-producer-network-thread | egress} 
org.apache.kafka.clients.producer.internals.Sender - Uncaught error in kafka 
producer I/O thread:
java.lang.IllegalStateException: Correlation id for response (90177743) does 
not match request (90177741)
at 
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:356)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:296)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:199)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Thread.java:745)
{code}

There were no producer errors preceding the sequence of IllegalStateExceptions. 
We have numerous other producers running and none of them reported the issue. 
The only solution was to terminate the process and restarting it.

Please let me know if you need any additional information.

> KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws 
> exception
> -
>
> Key: KAFKA-4669
> URL: https://issues.apache.org/jira/browse/KAFKA-4669
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Cheng Ju
>Priority: Critical
>  Labels: reliability
> Fix For: 0.10.3.0
>
>
> There is no try catch in NetworkClient.handleCompletedReceives.  If an 
> exception is thrown after inFlightRequests.completeNext(source), then the 
> corresponding RecordBatch's done will never get called, and 
> KafkaProducer.flush will hang on this RecordBatch.
> I've checked 0.10 code and think this bug does exist in 0.10 versions.
> A real case.  First a correlateId not match exception happens:
> 13 Jan 2017 17:08:24,059 ERROR [kafka-producer-network-thread | producer-21] 
> (org.apache.kafka.clients.producer.internals.Sender.run:130)  - Uncaught 
> error in kafka producer I/O thread: 
> java.lang.IllegalStateException: Correlation id for response (703766) does 
> not match request (703764)
>   at 
> org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:477)
>   at 
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:440)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
>   at java.lang.Thread.run(Thread.java:745)
> Then jstack shows the thread is hanging on:
>   at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>   at 
> org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:57)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:425)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:544)
>   at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:224)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
>   at java.lang.Thread.run(Thread.java:745)
> client code 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2609: KAFKA-4677: [Follow Up] add optimization to Sticky...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2609

KAFKA-4677: [Follow Up] add optimization to StickyTaskAssignor for rolling 
rebounce

Detect when a rebalance has happened due to one or more existing nodes 
bouncing. Keep assignment of previous active tasks the same and only assign the 
tasks that were not active to the new clients.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kstreams-575

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2609.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2609






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4677) Avoid unnecessary task movement across threads during rebalance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888161#comment-15888161
 ] 

ASF GitHub Bot commented on KAFKA-4677:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2609

KAFKA-4677: [Follow Up] add optimization to StickyTaskAssignor for rolling 
rebounce

Detect when a rebalance has happened due to one or more existing nodes 
bouncing. Keep assignment of previous active tasks the same and only assign the 
tasks that were not active to the new clients.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kstreams-575

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2609.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2609






> Avoid unnecessary task movement across threads during rebalance
> ---
>
> Key: KAFKA-4677
> URL: https://issues.apache.org/jira/browse/KAFKA-4677
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.3.0
>
>
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid 
> expensive task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus 
> we assume that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment 
> flips the assignment for task B and C (task A is newly assigned to either 
> thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for 
> B and C does change to the other thread resulting in unnecessary task 
> migration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

[Jeyhun Karimov]

Hi Eno,

Thanks for clarification. I think it is by definition allowed.  So if we
want to join a stream that uses wallclock time with a stream that uses an
event time, then we can assign the first one a timestamp extractor that
returns system clock, and for the second stream we can assign timestamp
extractor that extracts/computes the event time from record.

Cheers,
Jeyhun

On Tue, Feb 28, 2017 at 11:40 AM Eno Thereska 
wrote:

> Hi Jeyhun,
>
> I mean something slightly different. In your motivation you say "joining
> multiple streams/tables that require different timestamp extraction
> methods". I wan to understand the scope of this. Is it allowed to have a
> stream that uses wallclock time join a stream that uses event time? (It
> would be good to give some examples in the motivation about scenarios you
> envision). If the join is not allowed, how do you prevent that join from
> happening? Do you throw an exception?
>
> Thanks
> Eno
>
>
> > On 28 Feb 2017, at 10:04, Jeyhun Karimov  wrote:
> >
> > Hi Eno,
> >
> > Thanks for feedback. I think you mean [1]. In this KIP we do not consider
> > the situations you mentioned. So, either we can extend the KIP and solve
> > mentioned issues  or submit 2 PRs incrementally.
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-4785
> >
> >
> > Cheers,
> > Jeyhun
> >
> > On Tue, Feb 28, 2017 at 10:41 AM Eno Thereska 
> > wrote:
> >
> >> Hi Jeyhun,
> >>
> >> Thanks for the KIP, sorry I'm coming a bit late to the discussion.
> >>
> >> One thing I'd like to understand is whether we can avoid situations
> where
> >> the user is mixing different times (event time vs. wallclock time) in
> their
> >> processing inadvertently. Before this KIP, all the relevant topics have
> one
> >> time stamp extractor so that issue does not come up.
> >>
> >> What will be the behavior if times mismatch, e.g., for joins?
> >>
> >> Thanks
> >> Eno
> >>
> >>> On 22 Feb 2017, at 09:21, Jeyhun Karimov  wrote:
> >>>
> >>> Dear community,
> >>>
> >>> I would like to get further feedbacks on this KIP (if any).
> >>>
> >>> Cheers
> >>> Jeyhun
> >>>
> >>> On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax  >
> >>> wrote:
> >>>
>  Mathieu,
> 
>  I personally agree with your observation, and we have plans to submit
> a
>  KIP like this. If you want to drive this discussion feel free to start
>  the KIP by yourself!
> 
>  Having said that, for this KIP we might want to focus the discussion
> the
>  the actual feature that gets added: allowing to specify different
>  TS-Extractor for different inputs.
> 
> 
> 
>  -Matthias
> 
>  On 2/14/17 4:54 PM, Mathieu Fenniak wrote:
> > Hi Jeyhun,
> >
> > This KIP might not be the appropriate time, but my first thought
> >> reading
>  it
> > is that it might make sense to introduce a builder-style API rather
> >> than
> > adding a mix of new method overloads with independent optional
>  parameters.
> > :-)
> >
> > eg. stream(), table(), globalTable(), addSource(), could all accept a
> > "TopicReference" parameter that can be built like:
> >
> 
> >>
> TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build().
> >
> > Mathieu
> >
> >
> > On Tue, Feb 14, 2017 at 5:31 PM, Jeyhun Karimov <
> je.kari...@gmail.com>
> > wrote:
> >
> >> Dear community,
> >>
> >> I want to share the KIP-123 [1] which is based on issue KAFKA-4144
> >> [2].
>  You
> >> can check the PR in [3].
> >>
> >> I would like to get your comments.
> >>
> >> [1]
> >>
> 
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> >> [2] https://issues.apache.org/jira/browse/KAFKA-4144
> >> [3] https://github.com/apache/kafka/pull/2466
> >>
> >>
> >> Cheers,
> >> Jeyhun
> >> --
> >> -Cheers
> >>
> >> Jeyhun
> >>
> >
> 
>  --
> >>> -Cheers
> >>>
> >>> Jeyhun
> >>
> >> --
> > -Cheers
> >
> > Jeyhun
>
> --
-Cheers

Jeyhun

Like to contribute

[Sunitha Prabhu]

Hi

 I have been using for a while now and would like to be a contributor. I am
a core java developer.  May I know which part of kafka is written in java?
as it is also written in Scala.. Would you please help me pick up some jira
and add me to contributor list?

thanks
Sunitha

[jira] [Commented] (KAFKA-4811) ReplicaFetchThread may fail to create due to existing metric

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888339#comment-15888339
 ] 

Jun Rao commented on KAFKA-4811:


[~huxi_2b], yes, we should check both host and port and recreate the 
fetcherThread if either has changed.

> ReplicaFetchThread may fail to create due to existing metric
> 
>
> Key: KAFKA-4811
> URL: https://issues.apache.org/jira/browse/KAFKA-4811
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.10.2.0
>Reporter: Jun Rao
>Assignee: huxi
>  Labels: newbie
>
> Someone reported the following error.
> java.lang.IllegalArgumentException: A metric named 'MetricName 
> [name=connection-close-rate, group=replica-fetcher-metrics, 
> description=Connections closed per second in the window., tags={broker-id=1, 
> fetcher-id=0}]' already exists, can't register another one.
> at 
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:433)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:249)
> at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:234)
> at 
> org.apache.kafka.common.network.Selector$SelectorMetrics.(Selector.java:680)
> at org.apache.kafka.common.network.Selector.(Selector.java:140)
> at 
> kafka.server.ReplicaFetcherThread.(ReplicaFetcherThread.scala:86)
> at 
> kafka.server.ReplicaFetcherManager.createFetcherThread(ReplicaFetcherManager.scala:35)
> at 
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:83)
> at 
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:78)
> at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at 
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:78)
> at kafka.server.ReplicaManager.makeFollowers(ReplicaManager.scala:882)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:700)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:84)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:62)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl

[Stevo Slavic (JIRA)]

Stevo Slavic created KAFKA-4814:
---

 Summary: ZookeeperLeaderElector not respecting zookeeper.set.acl
 Key: KAFKA-4814
 URL: https://issues.apache.org/jira/browse/KAFKA-4814
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 0.10.1.1
Reporter: Stevo Slavic
Priority: Minor


By [migration guide|https://kafka.apache.org/documentation/#zk_authz_migration] 
for enabling ZooKeeper security on an existing Apache Kafka cluster, and 
[broker configuration 
documentation|https://kafka.apache.org/documentation/#brokerconfigs] for 
{{zookeeper.set.acl}} configuration property, when this property is set to 
false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even 
when JAAS config file is provisioned to broker. 

Problem is that there is broker side logic, like one in 
{{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}}, 
which does not respect this configuration property, resulting in ACLs being set 
even when there's just JAAS config file provisioned to Kafka broker while 
{{zookeeper.set.acl}} is set to {{false}}.

Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package of 
{{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only 
configuration property.

To make it possible without downtime to enable ZooKeeper authentication on 
existing cluster, it should be possible to have all Kafka brokers in cluster 
first authenticate to ZooKeeper cluster, without ACLs being set. Only once all 
ZooKeeper clients (Kafka brokers and others) are authenticating to ZooKeeper 
cluster then ACLs can be started being set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4814:
---
Fix Version/s: 0.10.2.1
   0.10.3.0

> ZookeeperLeaderElector not respecting zookeeper.set.acl
> ---
>
> Key: KAFKA-4814
> URL: https://issues.apache.org/jira/browse/KAFKA-4814
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.1.1
>Reporter: Stevo Slavic
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> By [migration 
> guide|https://kafka.apache.org/documentation/#zk_authz_migration] for 
> enabling ZooKeeper security on an existing Apache Kafka cluster, and [broker 
> configuration 
> documentation|https://kafka.apache.org/documentation/#brokerconfigs] for 
> {{zookeeper.set.acl}} configuration property, when this property is set to 
> false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even 
> when JAAS config file is provisioned to broker. 
> Problem is that there is broker side logic, like one in 
> {{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}}, 
> which does not respect this configuration property, resulting in ACLs being 
> set even when there's just JAAS config file provisioned to Kafka broker while 
> {{zookeeper.set.acl}} is set to {{false}}.
> Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package 
> of {{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only 
> configuration property.
> To make it possible without downtime to enable ZooKeeper authentication on 
> existing cluster, it should be possible to have all Kafka brokers in cluster 
> first authenticate to ZooKeeper cluster, without ACLs being set. Only once 
> all ZooKeeper clients (Kafka brokers and others) are authenticating to 
> ZooKeeper cluster then ACLs can be started being set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4814:
---
Labels: newbie  (was: )

> ZookeeperLeaderElector not respecting zookeeper.set.acl
> ---
>
> Key: KAFKA-4814
> URL: https://issues.apache.org/jira/browse/KAFKA-4814
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.1.1
>Reporter: Stevo Slavic
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> By [migration 
> guide|https://kafka.apache.org/documentation/#zk_authz_migration] for 
> enabling ZooKeeper security on an existing Apache Kafka cluster, and [broker 
> configuration 
> documentation|https://kafka.apache.org/documentation/#brokerconfigs] for 
> {{zookeeper.set.acl}} configuration property, when this property is set to 
> false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even 
> when JAAS config file is provisioned to broker. 
> Problem is that there is broker side logic, like one in 
> {{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}}, 
> which does not respect this configuration property, resulting in ACLs being 
> set even when there's just JAAS config file provisioned to Kafka broker while 
> {{zookeeper.set.acl}} is set to {{false}}.
> Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package 
> of {{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only 
> configuration property.
> To make it possible without downtime to enable ZooKeeper authentication on 
> existing cluster, it should be possible to have all Kafka brokers in cluster 
> first authenticate to ZooKeeper cluster, without ACLs being set. Only once 
> all ZooKeeper clients (Kafka brokers and others) are authenticating to 
> ZooKeeper cluster then ACLs can be started being set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4814) ZookeeperLeaderElector not respecting zookeeper.set.acl

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4814:
---
Priority: Major  (was: Minor)

> ZookeeperLeaderElector not respecting zookeeper.set.acl
> ---
>
> Key: KAFKA-4814
> URL: https://issues.apache.org/jira/browse/KAFKA-4814
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.10.1.1
>Reporter: Stevo Slavic
>  Labels: newbie
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> By [migration 
> guide|https://kafka.apache.org/documentation/#zk_authz_migration] for 
> enabling ZooKeeper security on an existing Apache Kafka cluster, and [broker 
> configuration 
> documentation|https://kafka.apache.org/documentation/#brokerconfigs] for 
> {{zookeeper.set.acl}} configuration property, when this property is set to 
> false Kafka brokers should not be setting any ACLs on ZooKeeper nodes, even 
> when JAAS config file is provisioned to broker. 
> Problem is that there is broker side logic, like one in 
> {{ZookeeperLeaderElector}} making use of {{JaasUtils#isZkSecurityEnabled}}, 
> which does not respect this configuration property, resulting in ACLs being 
> set even when there's just JAAS config file provisioned to Kafka broker while 
> {{zookeeper.set.acl}} is set to {{false}}.
> Notice that {{JaasUtils}} is in {{org.apache.kafka.common.security}} package 
> of {{kafka-clients}} module, while {{zookeeper.set.acl}} is broker side only 
> configuration property.
> To make it possible without downtime to enable ZooKeeper authentication on 
> existing cluster, it should be possible to have all Kafka brokers in cluster 
> first authenticate to ZooKeeper cluster, without ACLs being set. Only once 
> all ZooKeeper clients (Kafka brokers and others) are authenticating to 
> ZooKeeper cluster then ACLs can be started being set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Guozhang Wang]

＋1

On Tue, Feb 28, 2017 at 4:28 AM, Tom Crayford  wrote:

> +1 (non-binding)
>
> On Tue, 28 Feb 2017 at 11:42, Molnár Bálint 
> wrote:
>
> > +1
> >
> > 2017-02-28 12:17 GMT+01:00 Dongjin Lee :
> >
> > >
> > >
> > > +1.
> > >
> > >
> > >
> > > Best,
> > >
> > > Dongjin
> > >
> > >
> > >  --
> > >
> > >
> > >
> > >
> > > Dongjin Lee
> > >
> > >
> > >
> > >
> > >
> > > Software developer in Line+.
> > >
> > > So interested in massive-scale machine learning.
> > >
> > >
> > >
> > >  facebook:   www.facebook.com/dongjin.lee.kr (http://www.facebook.com/
> > > dongjin.lee.kr)
> > >
> > > linkedin:   kr.linkedin.com/in/dongjinleekr (
> http://kr.linkedin.com/in/
> > > dongjinleekr)
> > >
> > >
> > > github:   (http://goog_969573159/)github.com/dongjinleekr (
> > > http://github.com/dongjinleekr)
> > >
> > > twitter:   www.twitter.com/dongjinleekr (http://www.twitter.com/
> > > dongjinleekr)
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > >
> > > > On Feb 28, 2017 at 6:40 PM,  mailto:ism...@juma.me.uk
> )>
> > > wrote:
> > > >
> > > >
> > > >
> > > >  Hi everyone,
> > > >
> > > > Since the few who responded in the discuss thread were in favour and
> > > there
> > > > were no objections, I would like to initiate the voting process for
> > > > KIP-119: Drop Support for Scala 2.10 in Kafka 0.11:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
> > > >
> > > > The vote will run for a minimum of 72 hours.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > >
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-123: Allow per stream/table timestamp extractor

[Matthias J. Sax]

Eno,

I think this problem is out-of-scope and also present in the current
setting. We cannot avoid that a custom timestamp extractor uses and
if-else branch and returns different timestamps for different topic.
That is possible even right now.

Furthermore, the TimestampExtractor interface states:

> The extracted timestamp MUST represent the milliseconds since midnight, 
> January 1, 1970 UTC.

If uses don't follow this, there is nothing we can do about it.


-Matthias


On 2/28/17 7:47 AM, Jeyhun Karimov wrote:
> Hi Eno,
> 
> Thanks for clarification. I think it is by definition allowed.  So if we
> want to join a stream that uses wallclock time with a stream that uses an
> event time, then we can assign the first one a timestamp extractor that
> returns system clock, and for the second stream we can assign timestamp
> extractor that extracts/computes the event time from record.
> 
> Cheers,
> Jeyhun
> 
> On Tue, Feb 28, 2017 at 11:40 AM Eno Thereska 
> wrote:
> 
>> Hi Jeyhun,
>>
>> I mean something slightly different. In your motivation you say "joining
>> multiple streams/tables that require different timestamp extraction
>> methods". I wan to understand the scope of this. Is it allowed to have a
>> stream that uses wallclock time join a stream that uses event time? (It
>> would be good to give some examples in the motivation about scenarios you
>> envision). If the join is not allowed, how do you prevent that join from
>> happening? Do you throw an exception?
>>
>> Thanks
>> Eno
>>
>>
>>> On 28 Feb 2017, at 10:04, Jeyhun Karimov  wrote:
>>>
>>> Hi Eno,
>>>
>>> Thanks for feedback. I think you mean [1]. In this KIP we do not consider
>>> the situations you mentioned. So, either we can extend the KIP and solve
>>> mentioned issues  or submit 2 PRs incrementally.
>>>
>>> [1] https://issues.apache.org/jira/browse/KAFKA-4785
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>>
>>> On Tue, Feb 28, 2017 at 10:41 AM Eno Thereska 
>>> wrote:
>>>
 Hi Jeyhun,

 Thanks for the KIP, sorry I'm coming a bit late to the discussion.

 One thing I'd like to understand is whether we can avoid situations
>> where
 the user is mixing different times (event time vs. wallclock time) in
>> their
 processing inadvertently. Before this KIP, all the relevant topics have
>> one
 time stamp extractor so that issue does not come up.

 What will be the behavior if times mismatch, e.g., for joins?

 Thanks
 Eno

> On 22 Feb 2017, at 09:21, Jeyhun Karimov  wrote:
>
> Dear community,
>
> I would like to get further feedbacks on this KIP (if any).
>
> Cheers
> Jeyhun
>
> On Wed, Feb 15, 2017 at 2:36 AM Matthias J. Sax >>
> wrote:
>
>> Mathieu,
>>
>> I personally agree with your observation, and we have plans to submit
>> a
>> KIP like this. If you want to drive this discussion feel free to start
>> the KIP by yourself!
>>
>> Having said that, for this KIP we might want to focus the discussion
>> the
>> the actual feature that gets added: allowing to specify different
>> TS-Extractor for different inputs.
>>
>>
>>
>> -Matthias
>>
>> On 2/14/17 4:54 PM, Mathieu Fenniak wrote:
>>> Hi Jeyhun,
>>>
>>> This KIP might not be the appropriate time, but my first thought
 reading
>> it
>>> is that it might make sense to introduce a builder-style API rather
 than
>>> adding a mix of new method overloads with independent optional
>> parameters.
>>> :-)
>>>
>>> eg. stream(), table(), globalTable(), addSource(), could all accept a
>>> "TopicReference" parameter that can be built like:
>>>
>>

>> TopicReference("my-topic").keySerde(...).valueSerde(...).autoOffsetReset(...).timestampExtractor(...).build().
>>>
>>> Mathieu
>>>
>>>
>>> On Tue, Feb 14, 2017 at 5:31 PM, Jeyhun Karimov <
>> je.kari...@gmail.com>
>>> wrote:
>>>
 Dear community,

 I want to share the KIP-123 [1] which is based on issue KAFKA-4144
 [2].
>> You
 can check the PR in [3].

 I would like to get your comments.

 [1]

>>

>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
 [2] https://issues.apache.org/jira/browse/KAFKA-4144
 [3] https://github.com/apache/kafka/pull/2466


 Cheers,
 Jeyhun
 --
 -Cheers

 Jeyhun

>>>
>>
>> --
> -Cheers
>
> Jeyhun

 --
>>> -Cheers
>>>
>>> Jeyhun
>>
>> --
> -Cheers
> 
> Jeyhun
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Matthias J. Sax]

+1

Thanks a lot for the KIP!

-Matthias


On 2/28/17 1:35 AM, Damian Guy wrote:
> Thanks for the KIP Jeyhun!
> 
> +1
> 
> On Tue, 28 Feb 2017 at 08:59 Jeyhun Karimov  wrote:
> 
>> Dear community,
>>
>> I'd like to start the vote for KIP-123:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
>>
>>
>> Cheers,
>> Jeyhun
>> --
>> -Cheers
>>
>> Jeyhun
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Like to contribute

[Matthias J. Sax]

Sunitha,

Kafka Connect and Kafka Stream are both written in Java.

If you want to get started, look for Jira with label "beginner" or similar.

Also have a look here:
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes

Let us know if you have further question. Welcome to the community!


-Matthias

On 2/27/17 10:29 PM, Sunitha Prabhu wrote:
> Hi
> 
>  I have been using for a while now and would like to be a contributor. I am
> a core java developer.  May I know which part of kafka is written in java?
> as it is also written in Scala.. Would you please help me pick up some jira
> and add me to contributor list?
> 
> thanks
> Sunitha
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Jun Rao]

Hi, Dong,

RAID6 is an improvement over RAID5 and can tolerate 2 disks failure. Eno's
point is that the rebuild of RAID5/RAID6 requires reading more data
compared with RAID10, which increases the probability of error during
rebuild. This makes sense. In any case, do you think you could ask the SREs
at LinkedIn to share their opinions on RAID5/RAID6?

Yes, when a replica is offline due to a bad disk, it makes sense to handle
it immediately as if a StopReplicaRequest is received (i.e., replica is no
longer considered a leader and is removed from any replica fetcher thread).
Could you add that detail in item 2. in the wiki?

50. The wiki says "Broker assumes a log directory to be good after it
starts" : A log directory actually could be bad during startup.

51. In item 4, the wiki says "The controller watches the path
/log_dir_event_notification for new znode.". This doesn't seem be needed
now?

52. The isNewReplica field in LeaderAndIsrRequest should be for each
replica inside the replicas field, right?

Other than those, the current KIP looks good to me. Do you want to start a
separate discussion thread on KIP-113? I do have some comments there.

Thanks for working on this!

Jun


On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:

> Hi Jun,
>
> In addition to the Eno's reference of why rebuild time with RAID-5 is more
> expensive, another concern is that RAID-5 will fail if more than one disk
> fails. JBOD is still works with 1+ disk failure and has better performance
> with one disk failure. These seems like good argument for using JBOD
> instead of RAID-5.
>
> If a leader replica goes offline, the broker should first take all actions
> (i.e. remove the partition from fetcher thread) as if it has received
> StopReplicaRequest for this partition because the replica can no longer
> work anyway. It will also respond with error to any ProduceRequest and
> FetchRequest for partition. The broker notifies controller by writing
> notification znode in ZK. The controller learns the disk failure event from
> ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to learn
> that the replica is offline. The controller will then elect new leader for
> this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
> relevant brokers. The broker should stop adjusting the ISR for this
> partition as if the broker is already offline. I am not sure there is any
> inconsistency in broker's behavior when it is leader or follower. Is there
> any concern with this approach?
>
> Thanks for catching this. I have removed that reference from the KIP.
>
> Hi Eno,
>
> Thank you for providing the reference of the RAID-5. In LinkedIn we have 10
> disks per Kafka machine. It will not be a show-stopper operationally for
> LinkedIn if we have to deploy one-broker-per-disk. On the other hand we
> previously discussed the advantage of JBOD vs. one-broker-per-disk or
> one-broker-per-machine. One-broker-per-disk suffers from the problems
> described in the KIP and one-broker-per-machine increases the failure
> caused by disk failure by 10X. Since JBOD is strictly better than either of
> the two, it is also better then one-broker-per-multiple-disk which is
> somewhere between one-broker-per-disk and one-broker-per-machine.
>
> I personally think the benefits of JBOD design is worth the implementation
> complexity it introduces. I would also argue that it is reasonable for
> Kafka to manage this low level detail because Kafka is already exposing and
> managing replication factor of its data. But whether the complexity is
> worthwhile can be subjective and I can not prove my opinion. I am
> contributing significant amount of time to do this KIP because Kafka
> develops at LinkedIn believes it is useful and worth the effort. Yeah, it
> will be useful to see what everyone else think about it.
>
>
> Thanks,
> Dong
>
>
> On Mon, Feb 27, 2017 at 1:16 PM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > For RAID5, I am not sure the rebuild cost is a big concern. If a disk
> > fails, typically an admin has to bring down the broker, replace the
> failed
> > disk with a new one, trigger the RAID rebuild, and bring up the broker.
> > This way, there is no performance impact at runtime due to rebuild. The
> > benefit is that a broker doesn't fail in a hard way when there is a disk
> > failure and can be brought down in a controlled way for maintenance.
> While
> > the broker is running with a failed disk, reads may be more expensive
> since
> > they have to be computed from the parity. However, if most reads are from
> > page cache, this may not be a big issue either. So, it would be useful to
> > do some tests on RAID5 before we completely rule it out.
> >
> > Regarding whether to remove an offline replica from the fetcher thread
> > immediately. What do we do when a failed replica is a leader? Do we do
> > nothing or mark the replica as not the leader immediately? Intuitively,
> it
> > seems it's better if the broker acts consistently on a failed

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

[Jun Rao]

Hi, Dong,

Yes, this change makes sense to me.

Thanks,

Jun

On Mon, Feb 27, 2017 at 8:26 PM, Dong Lin  wrote:

> Hi Jun and everyone,
>
> I would like to change the KIP in the following way. Currently, if any
> replica if offline, the purge result for a partition will
> be NotEnoughReplicasException and its low_watermark will be 0. The
> motivation for this approach is that we want to guarantee that the data
> before purgedOffset has been deleted on all replicas of this partition if
> purge result indicates success.
>
> But this approach seems too conservative. It should be sufficient in most
> cases to just tell user success and set low_watermark to minimum
> logStartOffset of all live replicas in the PurgeResponse if logStartOffset
> of all live replicas have reached purgedOffset. This is because for an
> offline replicas to become online and be elected leader, it should have
> received one FetchReponse from the current leader which should tell it to
> purge beyond purgedOffset. The benefit of doing this change is that we can
> allow purge operation to succeed when some replica is offline.
>
> Are you OK with this change? If so, I will go ahead to update the KIP and
> implement this behavior.
>
> Thanks,
> Dong
>
>
>
> On Tue, Jan 17, 2017 at 10:18 AM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Do you have time to review the KIP again or vote for it?
> >
> > Hey Ewen,
> >
> > Can you also review the KIP again or vote for it? I have discussed with
> > Radai and Becket regarding your concern. We still think putting it in
> Admin
> > Client seems more intuitive because there is use-case where application
> > which manages topic or produces data may also want to purge data. It
> seems
> > weird if they need to create a consumer to do this.
> >
> > Thanks,
> > Dong
> >
> > On Thu, Jan 12, 2017 at 9:34 AM, Mayuresh Gharat <
> > gharatmayures...@gmail.com> wrote:
> >
> >> +1 (non-binding)
> >>
> >> Thanks,
> >>
> >> Mayuresh
> >>
> >> On Wed, Jan 11, 2017 at 1:03 PM, Dong Lin  wrote:
> >>
> >> > Sorry for the duplicated email. It seems that gmail will put the
> voting
> >> > email in this thread if I simply replace DISCUSS with VOTE in the
> >> subject.
> >> >
> >> > On Wed, Jan 11, 2017 at 12:57 PM, Dong Lin 
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > It seems that there is no further concern with the KIP-107. At this
> >> point
> >> > > we would like to start the voting process. The KIP can be found at
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107
> >> > > %3A+Add+purgeDataBefore%28%29+API+in+AdminClient.
> >> > >
> >> > > Thanks,
> >> > > Dong
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> -Regards,
> >> Mayuresh R. Gharat
> >> (862) 250-7125
> >>
> >
> >
>

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Jun Rao]

Hi, Ismael,

Good point on compatibility.

Hi, Mayuresh,

Given that, it seems that it's better to just add the raw principal as a
new field in Session for now and deprecate the KafkaPrincipal field in the
future if needed?

Thanks,

Jun

On Mon, Feb 27, 2017 at 5:05 PM, Ismael Juma  wrote:

> Breaking clients without a deprecation period is something we only do as a
> last resort. Is there strong justification for doing it here?
>
> Ismael
>
> On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
> > Hi Ismael,
> >
> > Yeah. I agree that it might break the clients if the user is using the
> > kafkaPrincipal directly. But since KafkaPrincipal is also a Java
> Principal
> > and I think, it would be a right thing to do replace the kafkaPrincipal
> > with Java Principal at this stage than later.
> >
> > We can mention in the KIP, that it would break the clients that are using
> > the KafkaPrincipal directly and they will have to use the PrincipalType
> > directly, if they are using it as its only one value and use the name
> from
> > the Principal directly or create a KafkaPrincipal from Java Principal as
> we
> > are doing in SimpleAclAuthorizer with this KIP.
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> >
> > On Mon, Feb 27, 2017 at 10:56 AM, Ismael Juma  wrote:
> >
> > > Hi Mayuresh,
> > >
> > > Sorry for the delay. The updated KIP states that there is no
> > compatibility
> > > impact, but that doesn't seem right. The fact that we changed the type
> of
> > > Session.principal to `Principal` means that any code that expects it to
> > be
> > > `KafkaPrincipal` will break. Either because of declared types (likely)
> or
> > > if it accesses `getPrincipalType` (unlikely since the value is always
> the
> > > same). It's a bit annoying, but we should add a new field to `Session`
> > with
> > > the original principal. We can potentially deprecate the existing one,
> if
> > > we're sure we don't need it (or we can leave it for now).
> > >
> > > Ismael
> > >
> > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com
> > > > wrote:
> > >
> > > > Hi Ismael, Joel, Becket
> > > >
> > > > Would you mind taking a look at this. We require 2 more binding votes
> > for
> > > > the KIP to pass.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Thu, Feb 23, 2017 at 10:57 AM, Dong Lin 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Wed, Feb 22, 2017 at 10:52 PM, Manikumar <
> > manikumar.re...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat <
> > > > > > gharatmayures...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks a lot for the comments and reviews.
> > > > > > > I agree we should log the username.
> > > > > > > What I meant by creating KafkaPrincipal was, after this KIP we
> > > would
> > > > > not
> > > > > > be
> > > > > > > required to create KafkaPrincipal and if we want to maintain
> the
> > > old
> > > > > > > logging, we will have to create it as we do today.
> > > > > > > I will take care that we specify the Principal name in the log.
> > > > > > >
> > > > > > > Thanks again for all the reviews.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mayuresh
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Jun Rao 
> > wrote:
> > > > > > >
> > > > > > > > Hi, Mayuresh,
> > > > > > > >
> > > > > > > > For logging the user name, we could do either way. We just
> need
> > > to
> > > > > make
> > > > > > > > sure the expected user name is logged. Also, currently, we
> are
> > > > > already
> > > > > > > > creating a KafkaPrincipal on every request. +1 on the latest
> > KIP.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat <
> > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks for the comments.
> > > > > > > > >
> > > > > > > > > I will mention in the KIP : how this change doesn't affect
> > the
> > > > > > default
> > > > > > > > > authorizer implementation.
> > > > > > > > >
> > > > > > > > > Regarding, Currently, we log the principal name in the
> > request
> > > > log
> > > > > in
> > > > > > > > > RequestChannel, which has the format of "principalType +
> > > > SEPARATOR
> > > > > +
> > > > > > > > > name;".
> > > > > > > > > It would be good if we can keep the same convention after
> > this
> > > > KIP.
> > > > > > One
> > > > > > > > way
> > > > > > > > > to do that is to convert java.security.Principal to
> > > > KafkaPrincipal
> > > > > > for
> > > > > > > > > logging the requests.
> > > > > > > > > --- > This would mean we have to create a new
> KafkaPrincipal
> > on
> > > > > each
> > > > > > > > > request. Would it

Re: [DISCUSS] KIP-124: Request rate quotas

[Colin McCabe]

I noticed that the throttle_time_ms added to all the message responses
is in milliseconds.  Does it make sense to express this in microseconds
in case we start doing more fine-grained CPU throttling later on?  An
int32 should still be more than enough if using microseconds.

best,
Colin


On Fri, Feb 24, 2017, at 10:31, Jun Rao wrote:
> Hi, Jay,
> 
> 2. Regarding request.unit vs request.percentage. I started with
> request.percentage too. The reasoning for request.unit is the following.
> Suppose that the capacity has been reached on a broker and the admin
> needs
> to add a new user. A simple way to increase the capacity is to increase
> the
> number of io threads, assuming there are still enough cores. If the limit
> is based on percentage, the additional capacity automatically gets
> distributed to existing users and we haven't really carved out any
> additional resource for the new user. Now, is it easy for a user to
> reason
> about 0.1 unit vs 10%. My feeling is that both are hard and have to be
> configured empirically. Not sure if percentage is obviously easier to
> reason about.
> 
> Thanks,
> 
> Jun
> 
> On Fri, Feb 24, 2017 at 8:10 AM, Jay Kreps  wrote:
> 
> > A couple of quick points:
> >
> > 1. Even though the implementation of this quota is only using io thread
> > time, i think we should call it something like "request-time". This will
> > give us flexibility to improve the implementation to cover network threads
> > in the future and will avoid exposing internal details like our thread
> > pools on the server.
> >
> > 2. Jun/Roger, I get what you are trying to fix but the idea of thread/units
> > is super unintuitive as a user-facing knob. I had to read the KIP like
> > eight times to understand this. I'm not sure that your point that
> > increasing the number of threads is a problem with a percentage-based
> > value, it really depends on whether the user thinks about the "percentage
> > of request processing time" or "thread units". If they think "I have
> > allocated 10% of my request processing time to user x" then it is a bug
> > that increasing the thread count decreases that percent as it does in the
> > current proposal. As a practical matter I think the only way to actually
> > reason about this is as a percent---I just don't believe people are going
> > to think, "ah, 4.3 thread units, that is the right amount!". Instead I
> > think they have to understand this thread unit concept, figure out what
> > they have set in number of threads, compute a percent and then come up with
> > the number of thread units, and these will all be wrong if that thread
> > count changes. I also think this ties us to throttling the I/O thread pool,
> > which may not be where we want to end up.
> >
> > 3. For what it's worth I do think having a single throttle_ms field in all
> > the responses that combines all throttling from all quotas is probably the
> > simplest. There could be a use case for having separate fields for each,
> > but I think that is actually harder to use/monitor in the common case so
> > unless someone has a use case I think just one should be fine.
> >
> > -Jay
> >
> > On Fri, Feb 24, 2017 at 4:21 AM, Rajini Sivaram 
> > wrote:
> >
> > > I have updated the KIP based on the discussions so far.
> > >
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Thu, Feb 23, 2017 at 11:29 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > Thank you all for the feedback.
> > > >
> > > > Ismael #1. It makes sense not to throttle inter-broker requests like
> > > > LeaderAndIsr etc. The simplest way to ensure that clients cannot use
> > > these
> > > > requests to bypass quotas for DoS attacks is to ensure that ACLs
> > prevent
> > > > clients from using these requests and unauthorized requests are
> > included
> > > > towards quotas.
> > > >
> > > > Ismael #2, Jay #1 : I was thinking that these quotas can return a
> > > separate
> > > > throttle time, and all utilization based quotas could use the same
> > field
> > > > (we won't add another one for network thread utilization for instance).
> > > But
> > > > perhaps it makes sense to keep byte rate quotas separate in
> > produce/fetch
> > > > responses to provide separate metrics? Agree with Ismael that the name
> > of
> > > > the existing field should be changed if we have two. Happy to switch
> > to a
> > > > single combined throttle time if that is sufficient.
> > > >
> > > > Ismael #4, #5, #6: Will update KIP. Will use dot separated name for new
> > > > property. Replication quotas use dot separated, so it will be
> > consistent
> > > > with all properties except byte rate quotas.
> > > >
> > > > Radai: #1 Request processing time rather than request rate were chosen
> > > > because the time per request can vary significantly between requests as
> > > > mentioned in the discussion and KIP.
> > > > #2 Two separate quotas for heartbeats/regular requests feel like more
> > > > configuration and more metrics. Since most

[GitHub] kafka pull request #2610: MINOR: Make asJsonSchema() and asConnectSchema() m...

[C0urante]

GitHub user C0urante opened a pull request:

https://github.com/apache/kafka/pull/2610

MINOR: Make asJsonSchema() and asConnectSchema() methods public

Want to use these methods in an external project.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/C0urante/kafka public-json-schema-conversion

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2610.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2610


commit 1f28682da64b7f088e741c9af7309406ca7eee2a
Author: Chris Egerton 
Date:   2017-02-28T17:30:54Z

Make asJsonSchema() and asConnectSchema() methods public




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Colin McCabe]

+1 (non-binding).

Thanks, Ismael.

cheers,
Colin


On Mon, Feb 27, 2017, at 19:47, Ismael Juma wrote:
> Hi all,
> 
> With 0.10.2.0 out of the way, I would like to volunteer to be the release
> manager for our next time-based release. See https://cwiki.apache.org/c
> onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> communication on time-based releases or need a reminder.
> 
> I put together a draft release plan with June 2017 as the release month
> (as
> previously agreed) and a list of KIPs that have already been voted:
> 
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68716876
> *
> 
> I haven't set exact dates for the various stages (feature freeze, code
> freeze, etc.) for now as Ewen is going to send out an email with some
> suggested tweaks based on his experience as release manager for 0.10.2.0.
> We can set the exact dates after that discussion.
> 
> As we are starting the process early this time, we should expect the
> number
> of KIPs in the plan to grow (so don't worry if your KIP is not there
> yet),
> but it's good to see that we already have 10 (including 2 merged and 2
> with
> PR reviews in progress).
> 
> Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> KIP-101
> (Leader Generation in Replication) require message format changes, which
> typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> it makes sense to also include KIP-106 (Unclean leader election should be
> false by default) and KIP-118 (Drop support for Java 7). We would also
> take
> the chance to remove deprecated code, in that case.
> 
> Given the above, how do people feel about 0.11.0.0 as the next Kafka
> version? Please share your thoughts.
> 
> Thanks,
> Ismael

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

[Mickael Maison]

Yes I agree, having a generic flag is more future proof.
I'll update the KIP in the coming days.

Thanks

On Tue, Feb 28, 2017 at 5:08 AM, Jason Gustafson  wrote:
> Hey Mickael,
>
> The suggestion to add something to Node makes sense. I could imagine for
> example adding a flag to indicate that the connection has a higher
> "priority," meaning that we can allocate outside of the memory pool if
> necessary. That would still be generic even if the only use case is the
> consumer coordinator. We might also face a similar problem when the
> producer is sending requests to the transaction coordinator for KIP-98.
> What do you think?
>
> Thanks,
> Jason
>
> On Mon, Feb 27, 2017 at 9:09 AM, Mickael Maison 
> wrote:
>
>> Apologies for the late response.
>>
>> Thanks Jason for the suggestion. Yes you are right, the Coordinator
>> connection is "tagged" with a different id, so we could retrieve it in
>> NetworkReceive to make the distinction.
>> However, currently the coordinator connection are made different by using:
>> Integer.MAX_VALUE - groupCoordinatorResponse.node().id()
>> for the Node id.
>>
>> So to identify Coordinator connections, we'd have to check that the
>> NetworkReceive source is a value near Integer.MAX_VALUE which is a bit
>> hacky ...
>>
>> Maybe we could add a constructor to Node that allows to pass in a
>> sourceId String. That way we could make all the coordinator
>> connections explicit (by setting it to "Coordinator-[ID]" for
>> example).
>> What do you think ?
>>
>> On Tue, Jan 24, 2017 at 12:58 AM, Jason Gustafson 
>> wrote:
>> > Good point. The consumer does use a separate connection to the
>> coordinator,
>> > so perhaps the connection itself could be tagged for normal heap
>> allocation?
>> >
>> > -Jason
>> >
>> > On Mon, Jan 23, 2017 at 10:26 AM, Onur Karaman <
>> onurkaraman.apa...@gmail.com
>> >> wrote:
>> >
>> >> I only did a quick scan but I wanted to point out what I think is an
>> >> incorrect assumption in the KIP's caveats:
>> >> "
>> >> There is a risk using the MemoryPool that, after we fill up the memory
>> with
>> >> fetch data, we can starve the coordinator's connection
>> >> ...
>> >> To alleviate this issue, only messages larger than 1Kb will be
>> allocated in
>> >> the MemoryPool. Smaller messages will be allocated directly on the Heap
>> >> like before. This allows group/heartbeat messages to avoid being
>> delayed if
>> >> the MemoryPool fills up.
>> >> "
>> >>
>> >> So it sounds like there's an incorrect assumption that responses from
>> the
>> >> coordinator will always be small (< 1Kb as mentioned in the caveat).
>> There
>> >> are now a handful of request types between clients and the coordinator:
>> >> {JoinGroup, SyncGroup, LeaveGroup, Heartbeat, OffsetCommit, OffsetFetch,
>> >> ListGroups, DescribeGroups}. While true (at least today) for
>> >> HeartbeatResponse and a few others, I don't think we can assume
>> >> JoinGroupResponse, SyncGroupResponse, DescribeGroupsResponse, and
>> >> OffsetFetchResponse will be small, as they are effectively bounded by
>> the
>> >> max message size allowed by the broker for the __consumer_offsets topic
>> >> which by default is 1MB.
>> >>
>> >> On Mon, Jan 23, 2017 at 9:46 AM, Mickael Maison <
>> mickael.mai...@gmail.com>
>> >> wrote:
>> >>
>> >> > I've updated the KIP to address all the comments raised here and from
>> >> > the "DISCUSS" thread.
>> >> > See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> >> > 81%3A+Bound+Fetch+memory+usage+in+the+consumer
>> >> >
>> >> > Now, I'd like to restart the vote.
>> >> >
>> >> > On Tue, Dec 6, 2016 at 9:02 AM, Rajini Sivaram
>> >> >  wrote:
>> >> > > Hi Mickael,
>> >> > >
>> >> > > I am +1 on the overall approach of this KIP, but have a couple of
>> >> > comments
>> >> > > (sorry, should have brought them up on the discuss thread earlier):
>> >> > >
>> >> > > 1. Perhaps it would be better to do this after KAFKA-4137
>> >> > >  is implemented?
>> At
>> >> > the
>> >> > > moment, coordinator shares the same NetworkClient (and hence the
>> same
>> >> > > Selector) with consumer connections used for fetching records. Since
>> >> > > freeing of memory relies on consuming applications invoking poll()
>> >> after
>> >> > > processing previous records and potentially after committing
>> offsets,
>> >> it
>> >> > > will be good to ensure that coordinator is not blocked for read by
>> >> fetch
>> >> > > responses. This may be simpler once coordinator has its own
>> Selector.
>> >> > >
>> >> > > 2. The KIP says: *Once messages are returned to the user, messages
>> are
>> >> > > deleted from the MemoryPool so new messages can be stored.*
>> >> > > Can you expand that a bit? I am assuming that partial buffers never
>> get
>> >> > > freed when some messages are returned to the user since the
>> consumer is
>> >> > > still holding a reference to the buffer. Would buffers be freed when
>> >> > > fetches for all the partitions in a response

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Mayuresh Gharat]

Hi Jun/Ismael,

Thanks for the comments.

I agree.
What I was thinking was, we get the KIP passed now and wait till major
kafka version release. We can then make this change, but for now we can
wait. Does that work?

If there are concerns, we can make the addition of extra field of type
Principal to Session and then deprecate the KafkaPrincipal later.

I am fine either ways. What do you think?

Thanks,

Mayuresh

On Tue, Feb 28, 2017 at 9:53 AM, Jun Rao  wrote:

> Hi, Ismael,
>
> Good point on compatibility.
>
> Hi, Mayuresh,
>
> Given that, it seems that it's better to just add the raw principal as a
> new field in Session for now and deprecate the KafkaPrincipal field in the
> future if needed?
>
> Thanks,
>
> Jun
>
> On Mon, Feb 27, 2017 at 5:05 PM, Ismael Juma  wrote:
>
> > Breaking clients without a deprecation period is something we only do as
> a
> > last resort. Is there strong justification for doing it here?
> >
> > Ismael
> >
> > On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com> wrote:
> >
> > > Hi Ismael,
> > >
> > > Yeah. I agree that it might break the clients if the user is using the
> > > kafkaPrincipal directly. But since KafkaPrincipal is also a Java
> > Principal
> > > and I think, it would be a right thing to do replace the kafkaPrincipal
> > > with Java Principal at this stage than later.
> > >
> > > We can mention in the KIP, that it would break the clients that are
> using
> > > the KafkaPrincipal directly and they will have to use the PrincipalType
> > > directly, if they are using it as its only one value and use the name
> > from
> > > the Principal directly or create a KafkaPrincipal from Java Principal
> as
> > we
> > > are doing in SimpleAclAuthorizer with this KIP.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > >
> > >
> > > On Mon, Feb 27, 2017 at 10:56 AM, Ismael Juma 
> wrote:
> > >
> > > > Hi Mayuresh,
> > > >
> > > > Sorry for the delay. The updated KIP states that there is no
> > > compatibility
> > > > impact, but that doesn't seem right. The fact that we changed the
> type
> > of
> > > > Session.principal to `Principal` means that any code that expects it
> to
> > > be
> > > > `KafkaPrincipal` will break. Either because of declared types
> (likely)
> > or
> > > > if it accesses `getPrincipalType` (unlikely since the value is always
> > the
> > > > same). It's a bit annoying, but we should add a new field to
> `Session`
> > > with
> > > > the original principal. We can potentially deprecate the existing
> one,
> > if
> > > > we're sure we don't need it (or we can leave it for now).
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat <
> > > > gharatmayures...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Ismael, Joel, Becket
> > > > >
> > > > > Would you mind taking a look at this. We require 2 more binding
> votes
> > > for
> > > > > the KIP to pass.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > > On Thu, Feb 23, 2017 at 10:57 AM, Dong Lin 
> > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 10:52 PM, Manikumar <
> > > manikumar.re...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat <
> > > > > > > gharatmayures...@gmail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks a lot for the comments and reviews.
> > > > > > > > I agree we should log the username.
> > > > > > > > What I meant by creating KafkaPrincipal was, after this KIP
> we
> > > > would
> > > > > > not
> > > > > > > be
> > > > > > > > required to create KafkaPrincipal and if we want to maintain
> > the
> > > > old
> > > > > > > > logging, we will have to create it as we do today.
> > > > > > > > I will take care that we specify the Principal name in the
> log.
> > > > > > > >
> > > > > > > > Thanks again for all the reviews.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Mayuresh
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Jun Rao 
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Mayuresh,
> > > > > > > > >
> > > > > > > > > For logging the user name, we could do either way. We just
> > need
> > > > to
> > > > > > make
> > > > > > > > > sure the expected user name is logged. Also, currently, we
> > are
> > > > > > already
> > > > > > > > > creating a KafkaPrincipal on every request. +1 on the
> latest
> > > KIP.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat <
> > > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks for the comments.
> > > > > > > > > >
> > > > > > > > > > I will mention in the KIP : how 

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Bill Bejeck]

+1

Sounds good to me. Thanks, Ismael.

-Bill

On Tue, Feb 28, 2017 at 1:01 PM, Colin McCabe  wrote:

> +1 (non-binding).
>
> Thanks, Ismael.
>
> cheers,
> Colin
>
>
> On Mon, Feb 27, 2017, at 19:47, Ismael Juma wrote:
> > Hi all,
> >
> > With 0.10.2.0 out of the way, I would like to volunteer to be the release
> > manager for our next time-based release. See https://cwiki.apache.org/c
> > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > communication on time-based releases or need a reminder.
> >
> > I put together a draft release plan with June 2017 as the release month
> > (as
> > previously agreed) and a list of KIPs that have already been voted:
> >
> > *https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68716876
> >  action?pageId=68716876>*
> >
> > I haven't set exact dates for the various stages (feature freeze, code
> > freeze, etc.) for now as Ewen is going to send out an email with some
> > suggested tweaks based on his experience as release manager for 0.10.2.0.
> > We can set the exact dates after that discussion.
> >
> > As we are starting the process early this time, we should expect the
> > number
> > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > yet),
> > but it's good to see that we already have 10 (including 2 merged and 2
> > with
> > PR reviews in progress).
> >
> > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > KIP-101
> > (Leader Generation in Replication) require message format changes, which
> > typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> > it makes sense to also include KIP-106 (Unclean leader election should be
> > false by default) and KIP-118 (Drop support for Java 7). We would also
> > take
> > the chance to remove deprecated code, in that case.
> >
> > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > version? Please share your thoughts.
> >
> > Thanks,
> > Ismael
>

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Todd Palino]

We have tested RAID 5/6 in the past (and recently) and found it to be
lacking. So, as noted, rebuild takes more time than RAID 10 because all the
disks need to be accessed to recalculate parity. In addition, there’s a
significant performance loss just in normal operations. It’s been a while
since I ran those tests, but it was in the 30-50% range - nothing to shrug
off. We didn’t even get to failure testing because of that.

Jun - to your question, we ran the tests with numerous combinations of
block sizes and FS parameters. The performance varied, but it was never
good enough to warrant more than a superficial look at using RAID 5/6. We
also tested both software RAID and hardware RAID.

As far as the operational concerns around broker-per-disk and
broker-per-server, we’ve been talking about this internally. Running one
broker per disk adds a good bit of administrative overhead and complexity.
If you perform a one by one rolling bounce of the cluster, you’re talking
about a 10x increase in time. That means a cluster that restarts in 30
minutes now takes 5 hours. If you try and optimize this by shutting down
all the brokers on one host at a time, you can get close to the original
number, but you now have added operational complexity by having to
micro-manage the bounce. The broker count increase will percolate down to
the rest of the administrative domain as well - maintaining ports for all
the instances, monitoring more instances, managing configs, etc.

You also have the overhead of running the extra processes - extra heap,
task switching, etc. We don’t have a problem with page cache really, since
the VM subsystem is fairly efficient about how it works. But just because
cache works doesn’t mean we’re not wasting other resources. And that gets
pushed downstream to clients as well, because they all have to maintain
more network connections and the resources that go along with it.

Running more brokers in a cluster also exposes you to more corner cases and
race conditions within the Kafka code. Bugs in the brokers, bugs in the
controllers, more complexity in balancing load in a cluster (though trying
to balance load across disks in a single broker doing JBOD negates that).

-Todd


On Tue, Feb 28, 2017 at 9:40 AM, Jun Rao  wrote:

> Hi, Dong,
>
> RAID6 is an improvement over RAID5 and can tolerate 2 disks failure. Eno's
> point is that the rebuild of RAID5/RAID6 requires reading more data
> compared with RAID10, which increases the probability of error during
> rebuild. This makes sense. In any case, do you think you could ask the SREs
> at LinkedIn to share their opinions on RAID5/RAID6?
>
> Yes, when a replica is offline due to a bad disk, it makes sense to handle
> it immediately as if a StopReplicaRequest is received (i.e., replica is no
> longer considered a leader and is removed from any replica fetcher thread).
> Could you add that detail in item 2. in the wiki?
>
> 50. The wiki says "Broker assumes a log directory to be good after it
> starts" : A log directory actually could be bad during startup.
>
> 51. In item 4, the wiki says "The controller watches the path
> /log_dir_event_notification for new znode.". This doesn't seem be needed
> now?
>
> 52. The isNewReplica field in LeaderAndIsrRequest should be for each
> replica inside the replicas field, right?
>
> Other than those, the current KIP looks good to me. Do you want to start a
> separate discussion thread on KIP-113? I do have some comments there.
>
> Thanks for working on this!
>
> Jun
>
>
> On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:
>
> > Hi Jun,
> >
> > In addition to the Eno's reference of why rebuild time with RAID-5 is
> more
> > expensive, another concern is that RAID-5 will fail if more than one disk
> > fails. JBOD is still works with 1+ disk failure and has better
> performance
> > with one disk failure. These seems like good argument for using JBOD
> > instead of RAID-5.
> >
> > If a leader replica goes offline, the broker should first take all
> actions
> > (i.e. remove the partition from fetcher thread) as if it has received
> > StopReplicaRequest for this partition because the replica can no longer
> > work anyway. It will also respond with error to any ProduceRequest and
> > FetchRequest for partition. The broker notifies controller by writing
> > notification znode in ZK. The controller learns the disk failure event
> from
> > ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to learn
> > that the replica is offline. The controller will then elect new leader
> for
> > this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
> > relevant brokers. The broker should stop adjusting the ISR for this
> > partition as if the broker is already offline. I am not sure there is any
> > inconsistency in broker's behavior when it is leader or follower. Is
> there
> > any concern with this approach?
> >
> > Thanks for catching this. I have removed that reference from the KIP.
> >
> > Hi Eno,
> >
> > Thank y

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hey Jun,

Certainly, I have added Todd to reply to the thread. And I have updated the
item to in the wiki.

50. The full statement is "Broker assumes a log directory to be good after
it starts, and mark log directory as bad once there is IOException when
broker attempts to access (i.e. read or write) the log directory". This
statement seems reasonable, right? If a log directory is actually bad, then
the broker will first assume it is OK, try to read logs on this log
directory, encounter IOException, and then mark it as bad.

51. My bad. I thought I removed it but I didn't. It is removed now.

52. I don't think so.. The isNewReplica field in the LeaderAndIsrRequest is
only relevant to the replica (i.e. broker) that receives the
LeaderAndIsrRequest. There is no need to specify whether each replica is
new inside LeaderAndIsrRequest. In other words, if a broker sends
LeaderAndIsrRequest to three different replicas of a given partition, the
isNewReplica field can be different across these three requests.

Yeah, I would definitely want to start discussion on KIP-113 after we have
reached agreement on KIP-112. I have actually opened KIP-113 discussion
thread on 1/12 together with this thread. I have yet to add the ability to
list offline directories in KIP-113 which we discussed in this thread.

Thanks for all your reviews! Is there further concern with the latest KIP?

Thanks!
Dong

On Tue, Feb 28, 2017 at 9:40 AM, Jun Rao  wrote:

> Hi, Dong,
>
> RAID6 is an improvement over RAID5 and can tolerate 2 disks failure. Eno's
> point is that the rebuild of RAID5/RAID6 requires reading more data
> compared with RAID10, which increases the probability of error during
> rebuild. This makes sense. In any case, do you think you could ask the SREs
> at LinkedIn to share their opinions on RAID5/RAID6?
>
> Yes, when a replica is offline due to a bad disk, it makes sense to handle
> it immediately as if a StopReplicaRequest is received (i.e., replica is no
> longer considered a leader and is removed from any replica fetcher thread).
> Could you add that detail in item 2. in the wiki?
>
> 50. The wiki says "Broker assumes a log directory to be good after it
> starts" : A log directory actually could be bad during startup.
>
> 51. In item 4, the wiki says "The controller watches the path
> /log_dir_event_notification for new znode.". This doesn't seem be needed
> now?
>
> 52. The isNewReplica field in LeaderAndIsrRequest should be for each
> replica inside the replicas field, right?
>
> Other than those, the current KIP looks good to me. Do you want to start a
> separate discussion thread on KIP-113? I do have some comments there.
>
> Thanks for working on this!
>
> Jun
>
>
> On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:
>
> > Hi Jun,
> >
> > In addition to the Eno's reference of why rebuild time with RAID-5 is
> more
> > expensive, another concern is that RAID-5 will fail if more than one disk
> > fails. JBOD is still works with 1+ disk failure and has better
> performance
> > with one disk failure. These seems like good argument for using JBOD
> > instead of RAID-5.
> >
> > If a leader replica goes offline, the broker should first take all
> actions
> > (i.e. remove the partition from fetcher thread) as if it has received
> > StopReplicaRequest for this partition because the replica can no longer
> > work anyway. It will also respond with error to any ProduceRequest and
> > FetchRequest for partition. The broker notifies controller by writing
> > notification znode in ZK. The controller learns the disk failure event
> from
> > ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to learn
> > that the replica is offline. The controller will then elect new leader
> for
> > this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
> > relevant brokers. The broker should stop adjusting the ISR for this
> > partition as if the broker is already offline. I am not sure there is any
> > inconsistency in broker's behavior when it is leader or follower. Is
> there
> > any concern with this approach?
> >
> > Thanks for catching this. I have removed that reference from the KIP.
> >
> > Hi Eno,
> >
> > Thank you for providing the reference of the RAID-5. In LinkedIn we have
> 10
> > disks per Kafka machine. It will not be a show-stopper operationally for
> > LinkedIn if we have to deploy one-broker-per-disk. On the other hand we
> > previously discussed the advantage of JBOD vs. one-broker-per-disk or
> > one-broker-per-machine. One-broker-per-disk suffers from the problems
> > described in the KIP and one-broker-per-machine increases the failure
> > caused by disk failure by 10X. Since JBOD is strictly better than either
> of
> > the two, it is also better then one-broker-per-multiple-disk which is
> > somewhere between one-broker-per-disk and one-broker-per-machine.
> >
> > I personally think the benefits of JBOD design is worth the
> implementation
> > complexity it introduces. I would also argue that it is r

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Eno Thereska]

Thanks Todd for the explanation.

Eno
> On 28 Feb 2017, at 18:15, Todd Palino  wrote:
> 
> We have tested RAID 5/6 in the past (and recently) and found it to be
> lacking. So, as noted, rebuild takes more time than RAID 10 because all the
> disks need to be accessed to recalculate parity. In addition, there’s a
> significant performance loss just in normal operations. It’s been a while
> since I ran those tests, but it was in the 30-50% range - nothing to shrug
> off. We didn’t even get to failure testing because of that.
> 
> Jun - to your question, we ran the tests with numerous combinations of
> block sizes and FS parameters. The performance varied, but it was never
> good enough to warrant more than a superficial look at using RAID 5/6. We
> also tested both software RAID and hardware RAID.
> 
> As far as the operational concerns around broker-per-disk and
> broker-per-server, we’ve been talking about this internally. Running one
> broker per disk adds a good bit of administrative overhead and complexity.
> If you perform a one by one rolling bounce of the cluster, you’re talking
> about a 10x increase in time. That means a cluster that restarts in 30
> minutes now takes 5 hours. If you try and optimize this by shutting down
> all the brokers on one host at a time, you can get close to the original
> number, but you now have added operational complexity by having to
> micro-manage the bounce. The broker count increase will percolate down to
> the rest of the administrative domain as well - maintaining ports for all
> the instances, monitoring more instances, managing configs, etc.
> 
> You also have the overhead of running the extra processes - extra heap,
> task switching, etc. We don’t have a problem with page cache really, since
> the VM subsystem is fairly efficient about how it works. But just because
> cache works doesn’t mean we’re not wasting other resources. And that gets
> pushed downstream to clients as well, because they all have to maintain
> more network connections and the resources that go along with it.
> 
> Running more brokers in a cluster also exposes you to more corner cases and
> race conditions within the Kafka code. Bugs in the brokers, bugs in the
> controllers, more complexity in balancing load in a cluster (though trying
> to balance load across disks in a single broker doing JBOD negates that).
> 
> -Todd
> 
> 
> On Tue, Feb 28, 2017 at 9:40 AM, Jun Rao  wrote:
> 
>> Hi, Dong,
>> 
>> RAID6 is an improvement over RAID5 and can tolerate 2 disks failure. Eno's
>> point is that the rebuild of RAID5/RAID6 requires reading more data
>> compared with RAID10, which increases the probability of error during
>> rebuild. This makes sense. In any case, do you think you could ask the SREs
>> at LinkedIn to share their opinions on RAID5/RAID6?
>> 
>> Yes, when a replica is offline due to a bad disk, it makes sense to handle
>> it immediately as if a StopReplicaRequest is received (i.e., replica is no
>> longer considered a leader and is removed from any replica fetcher thread).
>> Could you add that detail in item 2. in the wiki?
>> 
>> 50. The wiki says "Broker assumes a log directory to be good after it
>> starts" : A log directory actually could be bad during startup.
>> 
>> 51. In item 4, the wiki says "The controller watches the path
>> /log_dir_event_notification for new znode.". This doesn't seem be needed
>> now?
>> 
>> 52. The isNewReplica field in LeaderAndIsrRequest should be for each
>> replica inside the replicas field, right?
>> 
>> Other than those, the current KIP looks good to me. Do you want to start a
>> separate discussion thread on KIP-113? I do have some comments there.
>> 
>> Thanks for working on this!
>> 
>> Jun
>> 
>> 
>> On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:
>> 
>>> Hi Jun,
>>> 
>>> In addition to the Eno's reference of why rebuild time with RAID-5 is
>> more
>>> expensive, another concern is that RAID-5 will fail if more than one disk
>>> fails. JBOD is still works with 1+ disk failure and has better
>> performance
>>> with one disk failure. These seems like good argument for using JBOD
>>> instead of RAID-5.
>>> 
>>> If a leader replica goes offline, the broker should first take all
>> actions
>>> (i.e. remove the partition from fetcher thread) as if it has received
>>> StopReplicaRequest for this partition because the replica can no longer
>>> work anyway. It will also respond with error to any ProduceRequest and
>>> FetchRequest for partition. The broker notifies controller by writing
>>> notification znode in ZK. The controller learns the disk failure event
>> from
>>> ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to learn
>>> that the replica is offline. The controller will then elect new leader
>> for
>>> this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
>>> relevant brokers. The broker should stop adjusting the ISR for this
>>> partition as if the broker is already offline. I am not sure there is any
>>> 

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

[Vahid S Hashemian]

Thanks Jorge for addressing my suggestions. Looks good to me.

--Vahid



From:   Jorge Esteban Quilcate Otoya 
To: dev@kafka.apache.org
Date:   02/27/2017 01:57 AM
Subject:Re: KIP-122: Add a tool to Reset Consumer Group Offsets



@Vahid: make sense to add "new lag" info IMO, I will update the KIP.

@Becket:

1. About deleting, I think ConsumerGroupCommand already has an option to
delete Group information by topic. From delete docs: "Pass in groups to
delete topic partition offsets and ownership information over the entire
consumer group.". Let me know if this solves is enough for your case, of 
we
can consider to add something to the Reset Offsets tool.

2. Yes, for instance in the case of active consumers, the tool will
validate that there are no active consumers to avoid race conditions. I
have added some code snippets to the wiki, thanks for pointing that out.

El sáb., 25 feb. 2017 a las 0:29, Becket Qin ()
escribió:

> Thanks for the KIP Jorge. I think this is a useful KIP. I haven't read 
the
> KIP in detail yet, some comments from a quick review:
>
> 1. A glance at it it seems that there is no delete option. At LinkedIn 
we
> identified some cases that users want to delete the committed offset of 
a
> group. It would be good to include that as well.
>
> 2. It seems the KIP is missing some necessary implementation key points.
> e.g. how would the tool to commit offsets for a consumer group, does the
> broker need to know this is a special tool instead of an active consumer 
in
> the group (the generation check will be made on offset commit)? They are
> probably in your proof of concept code. Could you add them to the wiki 
as
> well?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Feb 24, 2017 at 1:19 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Thanks Jorge for addressing my question/suggestion.
> >
> > One last thing. I noticed is that in the example you have for the 
"plan"
> > option
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 122%3A+Add+Reset+Consumer+Group+Offsets+tooling#KIP-122:
> > AddResetConsumerGroupOffsetstooling-ExecutionOptions
> > )
> > under "Description" column, you put 0 for lag. So I assume that is the
> > current lag being reported, and not the new lag. Might be helpful to
> > explicitly specify that (i.e. CURRENT-LAG) in the column header.
> > The other option is to report both current and new lags, but I 
understand
> > if we don't want to do that since it's rather redundant info.
> >
> > Thanks again.
> > --Vahid
> >
> >
> >
> > From:   Jorge Esteban Quilcate Otoya 
> > To: dev@kafka.apache.org
> > Date:   02/24/2017 12:47 PM
> > Subject:Re: KIP-122: Add a tool to Reset Consumer Group 
Offsets
> >
> >
> >
> > Hi Vahid,
> >
> > Thanks for your comments. Check my answers below:
> >
> > El vie., 24 feb. 2017 a las 19:41, Vahid S Hashemian (<
> > vahidhashem...@us.ibm.com>) escribió:
> >
> > > Hi Jorge,
> > >
> > > Thanks for the useful KIP.
> > >
> > > I have a question regarding the proposed "plan" option.
> > > The "current offset" and "lag" values of a topic partition are
> > meaningful
> > > within a consumer group. In other words, different consumer groups
> could
> > > have different values for these properties of each topic partition.
> > > I don't see that reflected in the discussion around the "plan" 
option.
> > > Unless we are assuming a "--group" option is also provided by user
> > (which
> > > is not clear from the KIP if that is the case).
> > >
> >
> > I have added an additional comment to state that this options will
> require
> > a "group" argument.
> > It is considered to affect only one Consumer Group.
> >
> >
> > >
> > > Also, I was wondering if you can provide at least one full command
> > example
> > > for each of the "plan", "execute", and "export" options. They would
> > > definitely help in understanding some of the details.
> > >
> > >
> > Added to the KIP.
> >
> >
> > > Sorry for the delayed question/suggestion. I hope they make sense.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From:   Jorge Esteban Quilcate Otoya 
> > > To: dev@kafka.apache.org
> > > Date:   02/24/2017 09:51 AM
> > > Subject:Re: KIP-122: Add a tool to Reset Consumer Group 
Offsets
> > >
> > >
> > >
> > > Great! KIP updated.
> > >
> > >
> > >
> > > El vie., 24 feb. 2017 a las 18:22, Matthias J. Sax
> > > ()
> > > escribió:
> > >
> > > > I like this!
> > > >
> > > > --by-duration and --shift-by
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 2/24/17 12:57 AM, Jorge Esteban Quilcate Otoya wrote:
> > > > > Renaming to --by-duration LGTM
> > > > >
> > > > > Not sure about changing it to --shift-by-duration because we 
could
> > end
> > > up
> > > > > with the same redundancy as before with reset: --reset-offsets
> > > > > --reset-to-*.
> > > > >
> > > > > Maybe changing --shift-offset-by to --shift-by 'n' could make it
> > > > consistent
> > > > > enough?
> > > > >
> > > > >
> > > > > E

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

[Dong Lin]

Thanks Jun. I have updated the KIP to reflect this change.

On Tue, Feb 28, 2017 at 9:44 AM, Jun Rao  wrote:

> Hi, Dong,
>
> Yes, this change makes sense to me.
>
> Thanks,
>
> Jun
>
> On Mon, Feb 27, 2017 at 8:26 PM, Dong Lin  wrote:
>
> > Hi Jun and everyone,
> >
> > I would like to change the KIP in the following way. Currently, if any
> > replica if offline, the purge result for a partition will
> > be NotEnoughReplicasException and its low_watermark will be 0. The
> > motivation for this approach is that we want to guarantee that the data
> > before purgedOffset has been deleted on all replicas of this partition if
> > purge result indicates success.
> >
> > But this approach seems too conservative. It should be sufficient in most
> > cases to just tell user success and set low_watermark to minimum
> > logStartOffset of all live replicas in the PurgeResponse if
> logStartOffset
> > of all live replicas have reached purgedOffset. This is because for an
> > offline replicas to become online and be elected leader, it should have
> > received one FetchReponse from the current leader which should tell it to
> > purge beyond purgedOffset. The benefit of doing this change is that we
> can
> > allow purge operation to succeed when some replica is offline.
> >
> > Are you OK with this change? If so, I will go ahead to update the KIP and
> > implement this behavior.
> >
> > Thanks,
> > Dong
> >
> >
> >
> > On Tue, Jan 17, 2017 at 10:18 AM, Dong Lin  wrote:
> >
> > > Hey Jun,
> > >
> > > Do you have time to review the KIP again or vote for it?
> > >
> > > Hey Ewen,
> > >
> > > Can you also review the KIP again or vote for it? I have discussed with
> > > Radai and Becket regarding your concern. We still think putting it in
> > Admin
> > > Client seems more intuitive because there is use-case where application
> > > which manages topic or produces data may also want to purge data. It
> > seems
> > > weird if they need to create a consumer to do this.
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Thu, Jan 12, 2017 at 9:34 AM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >>
> > >> Mayuresh
> > >>
> > >> On Wed, Jan 11, 2017 at 1:03 PM, Dong Lin 
> wrote:
> > >>
> > >> > Sorry for the duplicated email. It seems that gmail will put the
> > voting
> > >> > email in this thread if I simply replace DISCUSS with VOTE in the
> > >> subject.
> > >> >
> > >> > On Wed, Jan 11, 2017 at 12:57 PM, Dong Lin 
> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > It seems that there is no further concern with the KIP-107. At
> this
> > >> point
> > >> > > we would like to start the voting process. The KIP can be found at
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107
> > >> > > %3A+Add+purgeDataBefore%28%29+API+in+AdminClient.
> > >> > >
> > >> > > Thanks,
> > >> > > Dong
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -Regards,
> > >> Mayuresh R. Gharat
> > >> (862) 250-7125
> > >>
> > >
> > >
> >
>

[GitHub] kafka pull request #2611: MINOR: improve MinTimestampTrackerTest and fix NPE...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2611

MINOR: improve MinTimestampTrackerTest and fix NPE when null element removed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka testing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2611.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2611


commit 7b702b9914c1b7f676f7f3d667e6ee2f0aef461d
Author: Damian Guy 
Date:   2017-02-28T17:53:45Z

improve MinTimestampTrackerTest and fix NPE when element with null removed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Joel Koshy]

If we deprecate KafkaPrincipal, then the Authorizer interface will also
need to change - i.e., deprecate the getAcls(KafkaPrincipal) method.

On Tue, Feb 28, 2017 at 10:11 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> Hi Jun/Ismael,
>
> Thanks for the comments.
>
> I agree.
> What I was thinking was, we get the KIP passed now and wait till major
> kafka version release. We can then make this change, but for now we can
> wait. Does that work?
>
> If there are concerns, we can make the addition of extra field of type
> Principal to Session and then deprecate the KafkaPrincipal later.
>
> I am fine either ways. What do you think?
>
> Thanks,
>
> Mayuresh
>
> On Tue, Feb 28, 2017 at 9:53 AM, Jun Rao  wrote:
>
> > Hi, Ismael,
> >
> > Good point on compatibility.
> >
> > Hi, Mayuresh,
> >
> > Given that, it seems that it's better to just add the raw principal as a
> > new field in Session for now and deprecate the KafkaPrincipal field in
> the
> > future if needed?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Feb 27, 2017 at 5:05 PM, Ismael Juma  wrote:
> >
> > > Breaking clients without a deprecation period is something we only do
> as
> > a
> > > last resort. Is there strong justification for doing it here?
> > >
> > > Ismael
> > >
> > > On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com> wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Yeah. I agree that it might break the clients if the user is using
> the
> > > > kafkaPrincipal directly. But since KafkaPrincipal is also a Java
> > > Principal
> > > > and I think, it would be a right thing to do replace the
> kafkaPrincipal
> > > > with Java Principal at this stage than later.
> > > >
> > > > We can mention in the KIP, that it would break the clients that are
> > using
> > > > the KafkaPrincipal directly and they will have to use the
> PrincipalType
> > > > directly, if they are using it as its only one value and use the name
> > > from
> > > > the Principal directly or create a KafkaPrincipal from Java Principal
> > as
> > > we
> > > > are doing in SimpleAclAuthorizer with this KIP.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > >
> > > >
> > > > On Mon, Feb 27, 2017 at 10:56 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Hi Mayuresh,
> > > > >
> > > > > Sorry for the delay. The updated KIP states that there is no
> > > > compatibility
> > > > > impact, but that doesn't seem right. The fact that we changed the
> > type
> > > of
> > > > > Session.principal to `Principal` means that any code that expects
> it
> > to
> > > > be
> > > > > `KafkaPrincipal` will break. Either because of declared types
> > (likely)
> > > or
> > > > > if it accesses `getPrincipalType` (unlikely since the value is
> always
> > > the
> > > > > same). It's a bit annoying, but we should add a new field to
> > `Session`
> > > > with
> > > > > the original principal. We can potentially deprecate the existing
> > one,
> > > if
> > > > > we're sure we don't need it (or we can leave it for now).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat <
> > > > > gharatmayures...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Ismael, Joel, Becket
> > > > > >
> > > > > > Would you mind taking a look at this. We require 2 more binding
> > votes
> > > > for
> > > > > > the KIP to pass.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > > On Thu, Feb 23, 2017 at 10:57 AM, Dong Lin 
> > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 10:52 PM, Manikumar <
> > > > manikumar.re...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat <
> > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jun,
> > > > > > > > >
> > > > > > > > > Thanks a lot for the comments and reviews.
> > > > > > > > > I agree we should log the username.
> > > > > > > > > What I meant by creating KafkaPrincipal was, after this KIP
> > we
> > > > > would
> > > > > > > not
> > > > > > > > be
> > > > > > > > > required to create KafkaPrincipal and if we want to
> maintain
> > > the
> > > > > old
> > > > > > > > > logging, we will have to create it as we do today.
> > > > > > > > > I will take care that we specify the Principal name in the
> > log.
> > > > > > > > >
> > > > > > > > > Thanks again for all the reviews.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Mayuresh
> > > > > > > > >
> > > > > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Jun Rao  >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Mayuresh,
> > > > > > > > > >
> > > > > > > > > > For logging the user name, we could do either way. We
> just
> > > need
> > > > > to
> > > > > > > make
> > > > > > > > > > sure the expected user name is logged. Also, currently,
> we

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Apurva Mehta]

+1 (non-binding) for 0.11.0

I do agree with Ismael's point that exactly-once should go through one
release of stabilization before bumping the version to 1.0.

Thanks,
Apurva

On Mon, Feb 27, 2017 at 7:47 PM, Ismael Juma  wrote:

> Hi all,
>
> With 0.10.2.0 out of the way, I would like to volunteer to be the release
> manager for our next time-based release. See https://cwiki.apache.org/c
> onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> communication on time-based releases or need a reminder.
>
> I put together a draft release plan with June 2017 as the release month (as
> previously agreed) and a list of KIPs that have already been voted:
>
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68716876
>  >*
>
> I haven't set exact dates for the various stages (feature freeze, code
> freeze, etc.) for now as Ewen is going to send out an email with some
> suggested tweaks based on his experience as release manager for 0.10.2.0.
> We can set the exact dates after that discussion.
>
> As we are starting the process early this time, we should expect the number
> of KIPs in the plan to grow (so don't worry if your KIP is not there yet),
> but it's good to see that we already have 10 (including 2 merged and 2 with
> PR reviews in progress).
>
> Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and KIP-101
> (Leader Generation in Replication) require message format changes, which
> typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> it makes sense to also include KIP-106 (Unclean leader election should be
> false by default) and KIP-118 (Drop support for Java 7). We would also take
> the chance to remove deprecated code, in that case.
>
> Given the above, how do people feel about 0.11.0.0 as the next Kafka
> version? Please share your thoughts.
>
> Thanks,
> Ismael
>

[GitHub] kafka pull request #2610: MINOR: Make asJsonSchema() and asConnectSchema() m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2610


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2484: KAFKA-3959: Follow-up; move upgrade notes to 0.10....

[ewencp]

Github user ewencp closed the pull request at:

https://github.com/apache/kafka/pull/2484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3959) __consumer_offsets wrong number of replicas at startup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888757#comment-15888757
 ] 

ASF GitHub Bot commented on KAFKA-3959:
---

Github user ewencp closed the pull request at:

https://github.com/apache/kafka/pull/2484


> __consumer_offsets wrong number of replicas at startup
> --
>
> Key: KAFKA-3959
> URL: https://issues.apache.org/jira/browse/KAFKA-3959
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, offset manager, replication
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.0.2, 0.10.1.0, 
> 0.10.1.1, 0.10.1.2
> Environment: Brokers of 3 kafka nodes running Red Hat Enterprise 
> Linux Server release 7.2 (Maipo)
>Reporter: Alban Hurtaud
>Assignee: Onur Karaman
>Priority: Blocker
>  Labels: needs-kip, reliability
> Fix For: 0.10.3.0
>
>
> When creating a stack of 3 kafka brokers, the consumer is starting faster 
> than kafka nodes and when trying to read a topic, only one kafka node is 
> available.
> So the __consumer_offsets is created with a replication factor set to 1 
> (instead of configured 3) :
> offsets.topic.replication.factor=3
> default.replication.factor=3
> min.insync.replicas=2
> Then, other kafka nodes go up and we have exceptions because the replicas # 
> for __consumer_offsets is 1 and min insync is 2. So exceptions are thrown.
> What I missed is : Why the __consumer_offsets is created with replication to 
> 1 (when 1 broker is running) whereas in server.properties it is set to 3 ?
> To reproduce : 
> - Prepare 3 kafka nodes with the 3 lines above added to servers.properties.
> - Run one kafka,
> - Run one consumer (the __consumer_offsets is created with replicas =1)
> - Run 2 more kafka nodes



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1307

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Make asJsonSchema() and asConnectSchema() methods public

--
[...truncated 7.43 KB...]
:328:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
offsetRetention + partitionData.timestamp
^
:575:
 method offsetData in class ListOffsetRequest is deprecated: see corresponding 
Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {
 ^
:575:
 class PartitionData in object ListOffsetRequest is deprecated: see 
corresponding Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {

  ^
:580:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  new ListOffsetResponse.PartitionData(Errors.UNKNOWN_TOPIC_OR_PARTITION, 
List[JLong]().asJava)
  ^
:605:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
(topicPartition, new ListOffsetResponse.PartitionData(Errors.NONE, 
offsets.map(new JLong(_)).asJava))
 ^
:612:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e), List[JLong]().asJava))
   ^
:615:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e), List[JLong]().asJava))
   ^
:269:
 class PartitionData in object ListOffsetRequest is deprecated: see 
corresponding Javadoc for more information.
val partitions = Map(topicPartition -> new 
ListOffsetRequest.PartitionData(earliestOrLatest, 1))
 ^
:280:
 value offsets in class PartitionData is deprecated: see corresponding Javadoc 
for more information.
  partitionData.offsets.get(0)
^
:45:
 class OldProducer in package producer is deprecated: This class has been 
deprecated and will be removed in a future release. Please use 
org.apache.kafka.clients.producer.KafkaProducer instead.
new OldProducer(getOldProducerProps(config))
^
:47:
 class NewShinyProducer in package producer is deprecated: This class has been 
deprecated and will be removed in a future release. Please use 
org.apache.kafka.clients.producer.KafkaProducer instead.
new NewShinyProducer(getNewProducerProps(config))
^
21 warnings found
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:kafka-trunk-jdk8:core:processResources UP-TO-DATE
:kafka-trunk-jdk8:core:classes
:kafka-trunk-jdk8:core:checkstyleMain
:kafka-trunk-jdk8:core:compileTestJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileTestScala
:88:
 method createAndShutdownStep in class MetricsTest is deprecated: This test has 
been deprecated and it will be removed in a future release
createAndShutdownStep("group0", "consumer0", "producer0")
^
one warning found
:kafka-trunk-jdk8:core:processTestResources
:kafka-trunk-jdk8:core:testClasses
:kafka-tr

[jira] [Commented] (KAFKA-4809) docker/run_tests.sh should set up /opt/kafka-dev to be the source directory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1509#comment-1509
 ] 

ASF GitHub Bot commented on KAFKA-4809:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2602


> docker/run_tests.sh should set up /opt/kafka-dev to be the source directory
> ---
>
> Key: KAFKA-4809
> URL: https://issues.apache.org/jira/browse/KAFKA-4809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>
> {{docker/run_tests.sh }} should be able to run 
> {{sanity_checks/test_verifiable_producer.py}} against version 0.8.2.2.  
> Currently, it can't do that, because the Docker image configures the contents 
> of {{/opt/kafka-dev}} differently than the Vagrant images are configured.  
> Currently, {{run_tests.sh}} decompresses the {{releaseTarGz}} in that 
> directory.  But it should simply be the source directory.  This would also 
> make it unnecessary to copy the {{releaseTarGz}} around.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2602: KAFKA-4809: docker/run_tests.sh should set up /opt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2602


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4809) docker/run_tests.sh should set up /opt/kafka-dev to be the source directory

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4809.
--
   Resolution: Fixed
 Reviewer: Ewen Cheslack-Postava
Fix Version/s: 0.10.2.1
   0.10.3.0

> docker/run_tests.sh should set up /opt/kafka-dev to be the source directory
> ---
>
> Key: KAFKA-4809
> URL: https://issues.apache.org/jira/browse/KAFKA-4809
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.10.3.0, 0.10.2.1
>
>
> {{docker/run_tests.sh }} should be able to run 
> {{sanity_checks/test_verifiable_producer.py}} against version 0.8.2.2.  
> Currently, it can't do that, because the Docker image configures the contents 
> of {{/opt/kafka-dev}} differently than the Vagrant images are configured.  
> Currently, {{run_tests.sh}} decompresses the {{releaseTarGz}} in that 
> directory.  But it should simply be the source directory.  This would also 
> make it unnecessary to copy the {{releaseTarGz}} around.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-82 - Add Record Headers

[radai]

I will settle for any API really, but just wanted to point out that as it
stands right now the API targets the most "advanced" (hence obscure and
rare) use cases, at the expense of the simple and common ones. i'd suggest
(as the minimal set):

Header header(String key) - returns JUST ONE (the very last) value given a
key
Iterable headers(String key) - returns ALL under a key
Iterable headers() - returns all, period. maybe allow null as key
to prev method instead?
void add(Header header) - appends a header (key inside).
void remove(String key) - removes ALL HEADERS under a key.

this way naive get/set semantics map to header(key)/add(Header) cleanly and
simply while preserving the ability to handle more advanced use cases.
we can always add more convenience methods (like those dealing with lists -
addAll etc) but i think the 5 (potentially 4) above are sufficient for
basically everything.

On Tue, Feb 28, 2017 at 4:08 AM, Ismael Juma  wrote:

> Hi Becket,
>
> Comments inline.
>
> On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin  wrote:
> >
> > 1. Regarding the mutability.
> >
> > I think it would be a big convenience to have headers mutable during
> > certain stage in the message life cycle for the use cases you mentioned.
> I
> > agree there is a material benefit especially given that we may have to
> > modify the headers for each message.
> >
> > That said, I also think it is fair to say that in the producer, in order
> to
> > guarantee the correctness of the entire logic, it is necessary that at
> some
> > point we need to make producer record immutable. For example we probably
> > don't want to see that users accidentally updated the headers when the
> > producer is doing the serialization or compression.
> >
> > Given that, would it be possible to make Headers to be able to switch
> from
> > mutable to immutable? We have done this for the Batch in the producer.
> For
> > example, initially the headers are mutable, but after it has gone through
> > all the interceptors, we can call Headers.close() to make it immutable
> > afterwards.
> >
>
> The difference is that the batch is an internal class that is not exposed
> to users. Can you please explain what happens if a user tries to send the
> same ProducerRecord twice? Would an interceptor fail when trying to mutate
> the header that is now closed? Or did you have something else in mind?
>
> Thanks,
> Ismael
>

[jira] [Created] (KAFKA-4815) Idempotent/transactional Producer Checklist (KIP-98)

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4815:
--

 Summary: Idempotent/transactional Producer Checklist (KIP-98)
 Key: KAFKA-4815
 URL: https://issues.apache.org/jira/browse/KAFKA-4815
 Project: Kafka
  Issue Type: New Feature
  Components: clients, core, producer 
Reporter: Jason Gustafson
Assignee: Jason Gustafson
 Fix For: 0.11.0.0


This issue tracks implementation progress for KIP-98: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Improve License Header Check

[Matthias J. Sax]

Just a reminder. This PR got merged today.


-Matthias

On 1/20/17 9:02 AM, Matthias J. Sax wrote:
> Hi,
> 
> I opened an PR to improve the check for file license header (the check
> is currently quite weak and it's possible to have files with an invalid
> header).
> 
> https://github.com/apache/kafka/pull/2303/
> 
> As some people do have IDE setting for adding a header automatically, we
> wanted to give a heads up that you will need to update you IDE setting.
> 
> 
> 
> -Matthias
> 
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-4816) Message format changes for idempotent/transactional producer

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4816:
--

 Summary: Message format changes for idempotent/transactional 
producer
 Key: KAFKA-4816
 URL: https://issues.apache.org/jira/browse/KAFKA-4816
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson


This task is for the implementation of the message format changes documented 
here: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging#KIP-98-ExactlyOnceDeliveryandTransactionalMessaging-MessageFormat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4817) Basic idempotent producer implementation

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4817:
--

 Summary: Basic idempotent producer implementation
 Key: KAFKA-4817
 URL: https://issues.apache.org/jira/browse/KAFKA-4817
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Apurva Mehta


This task covers the implementation of the idempotent producer for KIP-98. This 
covers both the necessary changes on the server-side and client-side changes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4738) Remove generic type of class ClientState

[Sharad (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1561#comment-1561
 ] 

Sharad commented on KAFKA-4738:
---

Yes, its done.

PR submitted:
https://github.com/apache/kafka/pull/2605

> Remove generic type of class ClientState
> 
>
> Key: KAFKA-4738
> URL: https://issues.apache.org/jira/browse/KAFKA-4738
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Sharad
>Priority: Minor
>  Labels: beginner, newbie
>
> Currently, class 
> {{org.apache.kafka.streams.processor.internals.assignment.ClientState}} 
> uses a generic type. However, within actual Streams code base the type will 
> always be {{TaskId}} (from package {{org.apache.kafka.streams.processor}}).
> Thus, this ticket is about removing the generic type and replace it with 
> {{TaskId}}, to simplify the code base.
> There are some tests, that use {{ClientState}} (what allows for a 
> slightly simplified test setup).  Those tests need to be updated to work 
> properly using {{TaskId}} instead of {{Integer}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2487: Kafka-4722 : Add application.id to StreamThread na...

[sharad-develop]

Github user sharad-develop closed the pull request at:

https://github.com/apache/kafka/pull/2487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4818) Implement transactional producer

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4818:
--

 Summary: Implement transactional producer
 Key: KAFKA-4818
 URL: https://issues.apache.org/jira/browse/KAFKA-4818
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Guozhang Wang


This covers the implementation of the transaction coordinator and the changes 
to the producer and consumer to support transactions. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4817) Implement idempotent producer

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4817:
---
Summary: Implement idempotent producer  (was: Basic idempotent producer 
implementation)

> Implement idempotent producer
> -
>
> Key: KAFKA-4817
> URL: https://issues.apache.org/jira/browse/KAFKA-4817
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> This task covers the implementation of the idempotent producer for KIP-98. 
> This covers both the necessary changes on the server-side and client-side 
> changes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4815) Idempotent/transactional Producer Checklist (KIP-98)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-4815:
--

Assignee: Matthias J. Sax  (was: Jason Gustafson)

> Idempotent/transactional Producer Checklist (KIP-98)
> 
>
> Key: KAFKA-4815
> URL: https://issues.apache.org/jira/browse/KAFKA-4815
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Matthias J. Sax
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> This issue tracks implementation progress for KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4815) Idempotent/transactional Producer Checklist (KIP-98)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-4815:
--

Assignee: (was: Matthias J. Sax)

> Idempotent/transactional Producer Checklist (KIP-98)
> 
>
> Key: KAFKA-4815
> URL: https://issues.apache.org/jira/browse/KAFKA-4815
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> This issue tracks implementation progress for KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4815) Idempotent/transactional Producer Checklist (KIP-98)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-4815:
--

Assignee: Jason Gustafson

> Idempotent/transactional Producer Checklist (KIP-98)
> 
>
> Key: KAFKA-4815
> URL: https://issues.apache.org/jira/browse/KAFKA-4815
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> This issue tracks implementation progress for KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4815) Idempotent/transactional Producer Checklist (KIP-98)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4815:
---
Labels: kip  (was: )

> Idempotent/transactional Producer Checklist (KIP-98)
> 
>
> Key: KAFKA-4815
> URL: https://issues.apache.org/jira/browse/KAFKA-4815
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Matthias J. Sax
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> This issue tracks implementation progress for KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1308

[Apache Jenkins Server]

See 


Changes:

[me] KAFKA-4809: docker/run_tests.sh should set up /opt/kafka-dev to be the

[me] MINOR: improve license header check by providing head file instead of

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-us1 (ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision d0e436c471ba4122ddcc0f7a1624546f97c4a517 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d0e436c471ba4122ddcc0f7a1624546f97c4a517
 > git rev-list 8e6fbe8fed592e7cc15731a0827c350794413767 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson845165826670901739.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 22.699 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7415865503829777297.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew --no-daemon -Dorg.gradle.project.maxParallelForks=1 
-Dorg.gradle.project.testLoggingEvents=started,passed,skipped,failed clean 
testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/3.2.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:connect:transforms:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:test_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJavawarning: [options] bootstrap class path 
not set in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar
:kafka-trunk-jdk8:clients:compileTestJavawarning: [options] bootstrap class 
path not set in conjunction with -source 1.7
1 warning

:kafka-trunk-jdk8:clients:processTestResources
:kafka-trunk-jdk8:clients:testClasses
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScala
:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.ka

[GitHub] kafka pull request #2487: Kafka-4722 : Add application.id to StreamThread na...

[sharad-develop]

GitHub user sharad-develop reopened a pull request:

https://github.com/apache/kafka/pull/2487

Kafka-4722 : Add application.id to StreamThread name

Kafka-4722 : Add application.id to StreamThread name

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sharad-develop/kafka KAFKA-4722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2487


commit 7946b2f35c0e3811ba7d706fc9c761d73ece47bb
Author: Damian Guy 
Date:   2017-01-16T19:40:47Z

MINOR: Remove unused constructor param from ProcessorStateManager

Remove applicationId parameter as it is no longer used.

Author: Damian Guy 

Reviewers: Guozhang Wang 

Closes #2385 from dguy/minor-remove-unused-param

commit 621dff22e79dc64b9a8748186dd985774044f91a
Author: Rajini Sivaram 
Date:   2017-01-17T11:16:29Z

KAFKA-4363; Documentation for sasl.jaas.config property

Author: Rajini Sivaram 

Reviewers: Ismael Juma 

Closes #2316 from rajinisivaram/KAFKA-4363

commit e3f4cdd0e249f78a7f4e8f064533bcd15eb11cbf
Author: Rajini Sivaram 
Date:   2017-01-17T12:55:07Z

KAFKA-4590; SASL/SCRAM system tests

Runs sanity test and one replication test using SASL/SCRAM.

Author: Rajini Sivaram 

Reviewers: Ewen Cheslack-Postava , Ismael Juma 


Closes #2355 from rajinisivaram/KAFKA-4590

commit 2b19ad9d8c47fb0f78a6e90d2f5711df6110bf1f
Author: Rajini Sivaram 
Date:   2017-01-17T18:42:55Z

KAFKA-4580; Use sasl.jaas.config for some system tests

Switched console_consumer, verifiable_consumer and verifiable_producer to 
use new sasl.jaas_config property instead of static JAAS configuration file 
when used with SASL_PLAINTEXT.

Author: Rajini Sivaram 

Reviewers: Ewen Cheslack-Postava , Ismael Juma 


Closes #2323 from rajinisivaram/KAFKA-4580

(cherry picked from commit 3f6c4f63c9c17424cf717ca76c74554bcf3b2e9a)
Signed-off-by: Ismael Juma 

commit 60d759a227087079e6fd270c68fd9e38441cb34a
Author: Jason Gustafson 
Date:   2017-01-17T18:42:05Z

MINOR: Some cleanups and additional testing for KIP-88

Author: Jason Gustafson 

Reviewers: Vahid Hashemian , Ismael Juma 


Closes #2383 from hachikuji/minor-cleanup-kip-88

commit c9b9acf6a8b542c2d0d825c17a4a20cf3fa5
Author: Damian Guy 
Date:   2017-01-17T20:33:11Z

KAFKA-4588: Wait for topics to be created in 
QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable

After debugging this i can see the times that it fails there is a race 
between when the topic is actually created/ready on the broker and when the 
assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets 
called with a `Cluster` with no topics. Hence the test hangs as no tasks get 
assigned. To fix this I added a `waitForTopics` method to 
`EmbeddedKafkaCluster`. This will wait until the topics have been created.

Author: Damian Guy 

Reviewers: Matthias J. Sax, Guozhang Wang

Closes #2371 from dguy/integration-test-fix

(cherry picked from commit 825f225bc5706b16af8ec44ca47ee1452c11e6f3)
Signed-off-by: Guozhang Wang 

commit 6f72a5a53c444278187fa6be58031168bcaffb26
Author: Damian Guy 
Date:   2017-01-17T22:13:46Z

KAFKA-3452 Follow-up: Refactoring StateStore hierarchies

This is a follow up of https://github.com/apache/kafka/pull/2166 - 
refactoring the store hierarchies as requested

Author: Damian Guy 

Reviewers: Guozhang Wang 

Closes #2360 from dguy/state-store-refactor

(cherry picked from commit 73b7ae0019d387407375f3865e263225c986a6ce)
Signed-off-by: Guozhang Wang 

commit eb62e5695506ae13bd37102c3c08e8a067eca0c8
Author: Ismael Juma 
Date:   2017-01-18T02:43:10Z

KAFKA-4591; Create Topic Policy follow-up

1. Added javadoc to public classes
2. Removed `s` from config name for consistency with interface name
3. The policy interface now implements Configurable and AutoCloseable as 
per the KIP
4. Use `null` instead of `-1` in `RequestMetadata`
5. Perform all broker validation before invoking the policy
6. Add tests

Author: Ismael Juma 

Reviewers: Jason Gustafson 

Closes #2388 from ijuma/create-topic-policy-docs-and-config-name-change

(cherry picked from commit fd6d7bcf335166a524dc9a29a50c96af8f1c1c02)
Signed-off-by: Ismael Juma 

commit e38794e020951adec5a5d0bbfe42c57294bf67bd
Author: Guozhang Wang 
Date:   2017-01-18T04:29:55Z

KAFKA-3502; move RocksDB options construction to init()

In RocksDBStore, options / wOptions / fOptions are constructed in the 
constructor, which needs to be dismissed 

[jira] [Created] (KAFKA-4819) Expose states of active tasks to public API

[Florian Hussonnois (JIRA)]

Florian Hussonnois created KAFKA-4819:
-

 Summary: Expose states of active tasks to public API
 Key: KAFKA-4819
 URL: https://issues.apache.org/jira/browse/KAFKA-4819
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Affects Versions: 0.10.2.0
Reporter: Florian Hussonnois
Priority: Minor


In Kafka 0.10.1.0 the toString method of KafkaStreams class has been 
implemented mainly to ease topologies debugging. Also,  the streams Metrics has 
been exposed to public API.

But currently theres is no way to monitor kstreams tasks states, assignments or 
consumed offsets.

I propose to expose the states of active tasks to the public API KafkaStreams.

For instance, an application can expose a REST API to get the global state of a 
kstreams topology.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2303: MINOR: improve license header check by providing h...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2303


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2607: MINOR: Fix typo in javadoc of `flatMapValues`

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2607


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2590: KAFKA-4789: Added support to ProcessorTopologyTest...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2590


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4789.
--
   Resolution: Fixed
Fix Version/s: 0.10.3.0

Issue resolved by pull request 2590
[https://github.com/apache/kafka/pull/2590]

> ProcessorTopologyTestDriver does not forward extracted timestamps to internal 
> topics
> 
>
> Key: KAFKA-4789
> URL: https://issues.apache.org/jira/browse/KAFKA-4789
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Hamidreza Afzali
>  Labels: unit-test
> Fix For: 0.10.3.0
>
>
> *Problem:*
> When using ProcessorTopologyTestDriver, the extracted timestamp is not 
> forwarded with the produced record to the internal topics.
> *Example:*
> {code}
> object Topology1 {
>   def main(args: Array[String]): Unit = {
> val inputTopic = "input"
> val outputTopic = "output"
> val stateStore = "count"
> val inputs = Seq[(String, Integer)](("A@145000", 1), ("B@145000", 
> 2))
> val props = new Properties
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, UUID.randomUUID().toString)
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
> classOf[MyTimestampExtractor].getName)
> val windowedStringSerde = Serdes.serdeFrom(new 
> WindowedSerializer(Serdes.String.serializer),
>   new WindowedDeserializer(Serdes.String.deserializer))
> val builder = new KStreamBuilder
> builder.stream(Serdes.String, Serdes.Integer, inputTopic)
>   .map[String, Integer]((k, v) => new KeyValue(k.split("@")(0), v))
>   .groupByKey(Serdes.String, Serdes.Integer)
>   .count(TimeWindows.of(1000L), stateStore)
>   .to(windowedStringSerde, Serdes.Long, outputTopic)
> val driver = new ProcessorTopologyTestDriver(new StreamsConfig(props), 
> builder, stateStore)
> inputs.foreach {
>   case (key, value) => {
> driver.process(inputTopic, key, value, Serdes.String.serializer, 
> Serdes.Integer.serializer)
> val record = driver.readOutput(outputTopic, 
> Serdes.String.deserializer, Serdes.Long.deserializer)
> println(record)
>   }
> }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888901#comment-15888901
 ] 

ASF GitHub Bot commented on KAFKA-4789:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2590


> ProcessorTopologyTestDriver does not forward extracted timestamps to internal 
> topics
> 
>
> Key: KAFKA-4789
> URL: https://issues.apache.org/jira/browse/KAFKA-4789
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Hamidreza Afzali
>  Labels: unit-test
> Fix For: 0.10.3.0
>
>
> *Problem:*
> When using ProcessorTopologyTestDriver, the extracted timestamp is not 
> forwarded with the produced record to the internal topics.
> *Example:*
> {code}
> object Topology1 {
>   def main(args: Array[String]): Unit = {
> val inputTopic = "input"
> val outputTopic = "output"
> val stateStore = "count"
> val inputs = Seq[(String, Integer)](("A@145000", 1), ("B@145000", 
> 2))
> val props = new Properties
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, UUID.randomUUID().toString)
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
> classOf[MyTimestampExtractor].getName)
> val windowedStringSerde = Serdes.serdeFrom(new 
> WindowedSerializer(Serdes.String.serializer),
>   new WindowedDeserializer(Serdes.String.deserializer))
> val builder = new KStreamBuilder
> builder.stream(Serdes.String, Serdes.Integer, inputTopic)
>   .map[String, Integer]((k, v) => new KeyValue(k.split("@")(0), v))
>   .groupByKey(Serdes.String, Serdes.Integer)
>   .count(TimeWindows.of(1000L), stateStore)
>   .to(windowedStringSerde, Serdes.Long, outputTopic)
> val driver = new ProcessorTopologyTestDriver(new StreamsConfig(props), 
> builder, stateStore)
> inputs.foreach {
>   case (key, value) => {
> driver.process(inputTopic, key, value, Serdes.String.serializer, 
> Serdes.Integer.serializer)
> val record = driver.readOutput(outputTopic, 
> Serdes.String.deserializer, Serdes.Long.deserializer)
> println(record)
>   }
> }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-4789:


Assignee: Hamidreza Afzali

> ProcessorTopologyTestDriver does not forward extracted timestamps to internal 
> topics
> 
>
> Key: KAFKA-4789
> URL: https://issues.apache.org/jira/browse/KAFKA-4789
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Hamidreza Afzali
>Assignee: Hamidreza Afzali
>  Labels: unit-test
> Fix For: 0.10.3.0
>
>
> *Problem:*
> When using ProcessorTopologyTestDriver, the extracted timestamp is not 
> forwarded with the produced record to the internal topics.
> *Example:*
> {code}
> object Topology1 {
>   def main(args: Array[String]): Unit = {
> val inputTopic = "input"
> val outputTopic = "output"
> val stateStore = "count"
> val inputs = Seq[(String, Integer)](("A@145000", 1), ("B@145000", 
> 2))
> val props = new Properties
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, UUID.randomUUID().toString)
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
> classOf[MyTimestampExtractor].getName)
> val windowedStringSerde = Serdes.serdeFrom(new 
> WindowedSerializer(Serdes.String.serializer),
>   new WindowedDeserializer(Serdes.String.deserializer))
> val builder = new KStreamBuilder
> builder.stream(Serdes.String, Serdes.Integer, inputTopic)
>   .map[String, Integer]((k, v) => new KeyValue(k.split("@")(0), v))
>   .groupByKey(Serdes.String, Serdes.Integer)
>   .count(TimeWindows.of(1000L), stateStore)
>   .to(windowedStringSerde, Serdes.Long, outputTopic)
> val driver = new ProcessorTopologyTestDriver(new StreamsConfig(props), 
> builder, stateStore)
> inputs.foreach {
>   case (key, value) => {
> driver.process(inputTopic, key, value, Serdes.String.serializer, 
> Serdes.Integer.serializer)
> val record = driver.readOutput(outputTopic, 
> Serdes.String.deserializer, Serdes.Long.deserializer)
> println(record)
>   }
> }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888903#comment-15888903
 ] 

Guozhang Wang commented on KAFKA-4789:
--

Thanks [~hrafzali] for the patch! I have added you to the contributor list so 
you can assign JIRAs to yourself in the future.

> ProcessorTopologyTestDriver does not forward extracted timestamps to internal 
> topics
> 
>
> Key: KAFKA-4789
> URL: https://issues.apache.org/jira/browse/KAFKA-4789
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Hamidreza Afzali
>Assignee: Hamidreza Afzali
>  Labels: unit-test
> Fix For: 0.10.3.0
>
>
> *Problem:*
> When using ProcessorTopologyTestDriver, the extracted timestamp is not 
> forwarded with the produced record to the internal topics.
> *Example:*
> {code}
> object Topology1 {
>   def main(args: Array[String]): Unit = {
> val inputTopic = "input"
> val outputTopic = "output"
> val stateStore = "count"
> val inputs = Seq[(String, Integer)](("A@145000", 1), ("B@145000", 
> 2))
> val props = new Properties
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, UUID.randomUUID().toString)
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
> classOf[MyTimestampExtractor].getName)
> val windowedStringSerde = Serdes.serdeFrom(new 
> WindowedSerializer(Serdes.String.serializer),
>   new WindowedDeserializer(Serdes.String.deserializer))
> val builder = new KStreamBuilder
> builder.stream(Serdes.String, Serdes.Integer, inputTopic)
>   .map[String, Integer]((k, v) => new KeyValue(k.split("@")(0), v))
>   .groupByKey(Serdes.String, Serdes.Integer)
>   .count(TimeWindows.of(1000L), stateStore)
>   .to(windowedStringSerde, Serdes.Long, outputTopic)
> val driver = new ProcessorTopologyTestDriver(new StreamsConfig(props), 
> builder, stateStore)
> inputs.foreach {
>   case (key, value) => {
> driver.process(inputTopic, key, value, Serdes.String.serializer, 
> Serdes.Integer.serializer)
> val record = driver.readOutput(outputTopic, 
> Serdes.String.deserializer, Serdes.Long.deserializer)
> println(record)
>   }
> }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Jun Rao]

Hi, Dong,

52. What you suggested would work. However, I am thinking that it's
probably simpler to just set isNewReplica at the replica level. That way,
the LeaderAndIsrRequest can be created a bit simpler. When reading a
LeaderAndIsrRequest in the controller log, it's easier to see which
replicas are new without looking at which broker the request is intended
for.

Could you also add those additional points from Todd's on 1 broker per disk
vs JBOD vs RAID5/6 to the KIP?

Thanks,

Hi, Todd,

Thanks for the feedback. That's very useful.

Jun

On Tue, Feb 28, 2017 at 10:25 AM, Dong Lin  wrote:

> Hey Jun,
>
> Certainly, I have added Todd to reply to the thread. And I have updated the
> item to in the wiki.
>
> 50. The full statement is "Broker assumes a log directory to be good after
> it starts, and mark log directory as bad once there is IOException when
> broker attempts to access (i.e. read or write) the log directory". This
> statement seems reasonable, right? If a log directory is actually bad, then
> the broker will first assume it is OK, try to read logs on this log
> directory, encounter IOException, and then mark it as bad.
>
> 51. My bad. I thought I removed it but I didn't. It is removed now.
>
> 52. I don't think so.. The isNewReplica field in the LeaderAndIsrRequest is
> only relevant to the replica (i.e. broker) that receives the
> LeaderAndIsrRequest. There is no need to specify whether each replica is
> new inside LeaderAndIsrRequest. In other words, if a broker sends
> LeaderAndIsrRequest to three different replicas of a given partition, the
> isNewReplica field can be different across these three requests.
>
> Yeah, I would definitely want to start discussion on KIP-113 after we have
> reached agreement on KIP-112. I have actually opened KIP-113 discussion
> thread on 1/12 together with this thread. I have yet to add the ability to
> list offline directories in KIP-113 which we discussed in this thread.
>
> Thanks for all your reviews! Is there further concern with the latest KIP?
>
> Thanks!
> Dong
>
> On Tue, Feb 28, 2017 at 9:40 AM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > RAID6 is an improvement over RAID5 and can tolerate 2 disks failure.
> Eno's
> > point is that the rebuild of RAID5/RAID6 requires reading more data
> > compared with RAID10, which increases the probability of error during
> > rebuild. This makes sense. In any case, do you think you could ask the
> SREs
> > at LinkedIn to share their opinions on RAID5/RAID6?
> >
> > Yes, when a replica is offline due to a bad disk, it makes sense to
> handle
> > it immediately as if a StopReplicaRequest is received (i.e., replica is
> no
> > longer considered a leader and is removed from any replica fetcher
> thread).
> > Could you add that detail in item 2. in the wiki?
> >
> > 50. The wiki says "Broker assumes a log directory to be good after it
> > starts" : A log directory actually could be bad during startup.
> >
> > 51. In item 4, the wiki says "The controller watches the path
> > /log_dir_event_notification for new znode.". This doesn't seem be needed
> > now?
> >
> > 52. The isNewReplica field in LeaderAndIsrRequest should be for each
> > replica inside the replicas field, right?
> >
> > Other than those, the current KIP looks good to me. Do you want to start
> a
> > separate discussion thread on KIP-113? I do have some comments there.
> >
> > Thanks for working on this!
> >
> > Jun
> >
> >
> > On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:
> >
> > > Hi Jun,
> > >
> > > In addition to the Eno's reference of why rebuild time with RAID-5 is
> > more
> > > expensive, another concern is that RAID-5 will fail if more than one
> disk
> > > fails. JBOD is still works with 1+ disk failure and has better
> > performance
> > > with one disk failure. These seems like good argument for using JBOD
> > > instead of RAID-5.
> > >
> > > If a leader replica goes offline, the broker should first take all
> > actions
> > > (i.e. remove the partition from fetcher thread) as if it has received
> > > StopReplicaRequest for this partition because the replica can no longer
> > > work anyway. It will also respond with error to any ProduceRequest and
> > > FetchRequest for partition. The broker notifies controller by writing
> > > notification znode in ZK. The controller learns the disk failure event
> > from
> > > ZK, sends LeaderAndIsrRequest and receives LeaderAndIsrResponse to
> learn
> > > that the replica is offline. The controller will then elect new leader
> > for
> > > this partition and sends LeaderAndIsrRequest/MetadataUpdateRequest to
> > > relevant brokers. The broker should stop adjusting the ISR for this
> > > partition as if the broker is already offline. I am not sure there is
> any
> > > inconsistency in broker's behavior when it is leader or follower. Is
> > there
> > > any concern with this approach?
> > >
> > > Thanks for catching this. I have removed that reference from the KIP.
> > >
> > > Hi Eno,
> > >
> > > Thank you for prov

[GitHub] kafka pull request #2612: KAFKA-4819: Expose states for active tasks to publ...

[fhussonnois]

GitHub user fhussonnois opened a pull request:

https://github.com/apache/kafka/pull/2612

KAFKA-4819: Expose states for active tasks to public API

Simple implementation of the feature : 
[KAFKA-4819](https://issues.apache.org/jira/browse/KAFKA-4819)
 KAFKA-4819

This PR adds a new method `threadStates` to public API of `KafkaStreams` 
which returns all currently states of running threads and active tasks.

Below is a example for a simple topology consuming from topics; test-p2 and 
test-p4.

[{"name":"StreamThread-1","state":"RUNNING","activeTasks":[{"id":"0_0", 
"assignments":["test-p4-0","test-p2-0"], 
"consumedOffsetsByPartition":[{"topicPartition":"test-p2-0","offset":"test-p2-0"}]},
 {"id":"0_2", "assignments":["test-p4-2"], "consumedOffsetsByPartition":[]}]}, 
{"name":"StreamThread-2","state":"RUNNING","activeTasks":[{"id":"0_1", 
"assignments":["test-p4-1","test-p2-1"], 
"consumedOffsetsByPartition":[{"topicPartition":"test-p2-1","offset":"test-p2-1"}]},
 {"id":"0_3", "assignments":["test-p4-3"], "consumedOffsetsByPartition":[]}]}]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhussonnois/kafka KAFKA-4819

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2612.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2612


commit 0f8b8123cabdbfcfb44fe59b9be20e13ac253c95
Author: Florian Hussonnois 
Date:   2017-02-23T22:08:01Z

KAFKA-4819: Expose states for active tasks to public API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4819) Expose states of active tasks to public API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888916#comment-15888916
 ] 

ASF GitHub Bot commented on KAFKA-4819:
---

GitHub user fhussonnois opened a pull request:

https://github.com/apache/kafka/pull/2612

KAFKA-4819: Expose states for active tasks to public API

Simple implementation of the feature : 
[KAFKA-4819](https://issues.apache.org/jira/browse/KAFKA-4819)
 KAFKA-4819

This PR adds a new method `threadStates` to public API of `KafkaStreams` 
which returns all currently states of running threads and active tasks.

Below is a example for a simple topology consuming from topics; test-p2 and 
test-p4.

[{"name":"StreamThread-1","state":"RUNNING","activeTasks":[{"id":"0_0", 
"assignments":["test-p4-0","test-p2-0"], 
"consumedOffsetsByPartition":[{"topicPartition":"test-p2-0","offset":"test-p2-0"}]},
 {"id":"0_2", "assignments":["test-p4-2"], "consumedOffsetsByPartition":[]}]}, 
{"name":"StreamThread-2","state":"RUNNING","activeTasks":[{"id":"0_1", 
"assignments":["test-p4-1","test-p2-1"], 
"consumedOffsetsByPartition":[{"topicPartition":"test-p2-1","offset":"test-p2-1"}]},
 {"id":"0_3", "assignments":["test-p4-3"], "consumedOffsetsByPartition":[]}]}]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fhussonnois/kafka KAFKA-4819

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2612.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2612


commit 0f8b8123cabdbfcfb44fe59b9be20e13ac253c95
Author: Florian Hussonnois 
Date:   2017-02-23T22:08:01Z

KAFKA-4819: Expose states for active tasks to public API




> Expose states of active tasks to public API
> ---
>
> Key: KAFKA-4819
> URL: https://issues.apache.org/jira/browse/KAFKA-4819
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Florian Hussonnois
>Priority: Minor
>
> In Kafka 0.10.1.0 the toString method of KafkaStreams class has been 
> implemented mainly to ease topologies debugging. Also,  the streams Metrics 
> has been exposed to public API.
> But currently theres is no way to monitor kstreams tasks states, assignments 
> or consumed offsets.
> I propose to expose the states of active tasks to the public API KafkaStreams.
> For instance, an application can expose a REST API to get the global state of 
> a kstreams topology.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

[Jun Rao]

Hi, Joel,

Good point on the getAcls() method. KafkaPrincipal is also tied to ACL,
which is used in pretty much every method in Authorizer. Now, I am not sure
if it's easy to deprecate KafkaPrincipal.

Hi, Mayuresh,

Given the above, it seems that the easiest thing is to add a new Principal
field in Session. We want to make it clear that it's ignored in the default
implementation, but a customizer authorizer could take advantage of that.

Thanks,

Jun

On Tue, Feb 28, 2017 at 10:52 AM, Joel Koshy  wrote:

> If we deprecate KafkaPrincipal, then the Authorizer interface will also
> need to change - i.e., deprecate the getAcls(KafkaPrincipal) method.
>
> On Tue, Feb 28, 2017 at 10:11 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com> wrote:
>
> > Hi Jun/Ismael,
> >
> > Thanks for the comments.
> >
> > I agree.
> > What I was thinking was, we get the KIP passed now and wait till major
> > kafka version release. We can then make this change, but for now we can
> > wait. Does that work?
> >
> > If there are concerns, we can make the addition of extra field of type
> > Principal to Session and then deprecate the KafkaPrincipal later.
> >
> > I am fine either ways. What do you think?
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Tue, Feb 28, 2017 at 9:53 AM, Jun Rao  wrote:
> >
> > > Hi, Ismael,
> > >
> > > Good point on compatibility.
> > >
> > > Hi, Mayuresh,
> > >
> > > Given that, it seems that it's better to just add the raw principal as
> a
> > > new field in Session for now and deprecate the KafkaPrincipal field in
> > the
> > > future if needed?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Feb 27, 2017 at 5:05 PM, Ismael Juma 
> wrote:
> > >
> > > > Breaking clients without a deprecation period is something we only do
> > as
> > > a
> > > > last resort. Is there strong justification for doing it here?
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat <
> > > > gharatmayures...@gmail.com> wrote:
> > > >
> > > > > Hi Ismael,
> > > > >
> > > > > Yeah. I agree that it might break the clients if the user is using
> > the
> > > > > kafkaPrincipal directly. But since KafkaPrincipal is also a Java
> > > > Principal
> > > > > and I think, it would be a right thing to do replace the
> > kafkaPrincipal
> > > > > with Java Principal at this stage than later.
> > > > >
> > > > > We can mention in the KIP, that it would break the clients that are
> > > using
> > > > > the KafkaPrincipal directly and they will have to use the
> > PrincipalType
> > > > > directly, if they are using it as its only one value and use the
> name
> > > > from
> > > > > the Principal directly or create a KafkaPrincipal from Java
> Principal
> > > as
> > > > we
> > > > > are doing in SimpleAclAuthorizer with this KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2017 at 10:56 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Hi Mayuresh,
> > > > > >
> > > > > > Sorry for the delay. The updated KIP states that there is no
> > > > > compatibility
> > > > > > impact, but that doesn't seem right. The fact that we changed the
> > > type
> > > > of
> > > > > > Session.principal to `Principal` means that any code that expects
> > it
> > > to
> > > > > be
> > > > > > `KafkaPrincipal` will break. Either because of declared types
> > > (likely)
> > > > or
> > > > > > if it accesses `getPrincipalType` (unlikely since the value is
> > always
> > > > the
> > > > > > same). It's a bit annoying, but we should add a new field to
> > > `Session`
> > > > > with
> > > > > > the original principal. We can potentially deprecate the existing
> > > one,
> > > > if
> > > > > > we're sure we don't need it (or we can leave it for now).
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat <
> > > > > > gharatmayures...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Ismael, Joel, Becket
> > > > > > >
> > > > > > > Would you mind taking a look at this. We require 2 more binding
> > > votes
> > > > > for
> > > > > > > the KIP to pass.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mayuresh
> > > > > > >
> > > > > > > On Thu, Feb 23, 2017 at 10:57 AM, Dong Lin <
> lindon...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 10:52 PM, Manikumar <
> > > > > manikumar.re...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat <
> > > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks a lot for the comments and reviews.
> > > > > > > > > > I agree we should log the username.
> > > > > > > > > > What I meant by creating KafkaPrincipal was, after this
> KIP
> > > we
> > > >

[DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

[Guozhang Wang]

Hi all,

I have just created KIP-129 to leverage KIP-98 in Kafka Streams and provide
exactly-once processing semantics:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics

This KIP enables Streams users to optionally turn on exactly-once
processing semantics without changing their app code at all by leveraging
the transactional messaging features provided in KIP-98.

The above wiki page provides a high-level view of the proposed changes,
while detailed implementation design can be found in this Google doc:

https://docs.google.com/document/d/1pGZ8xtOOyGwDYgH5vA6h19zOMMaduFK1DAB8_gBYA2c

We would love to hear your comments and suggestions.

Thanks,
-- Guozhang

[jira] [Commented] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[Hamidreza Afzali (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888941#comment-15888941
 ] 

Hamidreza Afzali commented on KAFKA-4789:
-

Thanks!

> ProcessorTopologyTestDriver does not forward extracted timestamps to internal 
> topics
> 
>
> Key: KAFKA-4789
> URL: https://issues.apache.org/jira/browse/KAFKA-4789
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Hamidreza Afzali
>Assignee: Hamidreza Afzali
>  Labels: unit-test
> Fix For: 0.10.3.0
>
>
> *Problem:*
> When using ProcessorTopologyTestDriver, the extracted timestamp is not 
> forwarded with the produced record to the internal topics.
> *Example:*
> {code}
> object Topology1 {
>   def main(args: Array[String]): Unit = {
> val inputTopic = "input"
> val outputTopic = "output"
> val stateStore = "count"
> val inputs = Seq[(String, Integer)](("A@145000", 1), ("B@145000", 
> 2))
> val props = new Properties
> props.put(StreamsConfig.APPLICATION_ID_CONFIG, UUID.randomUUID().toString)
> props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
> props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
> classOf[MyTimestampExtractor].getName)
> val windowedStringSerde = Serdes.serdeFrom(new 
> WindowedSerializer(Serdes.String.serializer),
>   new WindowedDeserializer(Serdes.String.deserializer))
> val builder = new KStreamBuilder
> builder.stream(Serdes.String, Serdes.Integer, inputTopic)
>   .map[String, Integer]((k, v) => new KeyValue(k.split("@")(0), v))
>   .groupByKey(Serdes.String, Serdes.Integer)
>   .count(TimeWindows.of(1000L), stateStore)
>   .to(windowedStringSerde, Serdes.Long, outputTopic)
> val driver = new ProcessorTopologyTestDriver(new StreamsConfig(props), 
> builder, stateStore)
> inputs.foreach {
>   case (key, value) => {
> driver.process(inputTopic, key, value, Serdes.String.serializer, 
> Serdes.Integer.serializer)
> val record = driver.readOutput(outputTopic, 
> Serdes.String.deserializer, Serdes.Long.deserializer)
> println(record)
>   }
> }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4738) Remove generic type of class ClientState

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888950#comment-15888950
 ] 

Matthias J. Sax commented on KAFKA-4738:


[~sharad.develop] Your PR shows 81 commits. We cannot review/merge it like 
this. Please rebase your PR to current trunk to get rid of all the duplicate 
commits.

> Remove generic type of class ClientState
> 
>
> Key: KAFKA-4738
> URL: https://issues.apache.org/jira/browse/KAFKA-4738
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Sharad
>Priority: Minor
>  Labels: beginner, newbie
>
> Currently, class 
> {{org.apache.kafka.streams.processor.internals.assignment.ClientState}} 
> uses a generic type. However, within actual Streams code base the type will 
> always be {{TaskId}} (from package {{org.apache.kafka.streams.processor}}).
> Thus, this ticket is about removing the generic type and replace it with 
> {{TaskId}}, to simplify the code base.
> There are some tests, that use {{ClientState}} (what allows for a 
> slightly simplified test setup).  Those tests need to be updated to work 
> properly using {{TaskId}} instead of {{Integer}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hi Jun,

Yeah there is tradeoff between controller's implementation complexity vs.
wire-protocol complexity. I personally think it is more important to keep
wire-protocol concise and only add information in wire-protocol if
necessary. It seems fine to add a little bit complexity to controller's
implementation, e.g. log destination broker per LeaderAndIsrRequet. Becket
also shares this opinion with me. Is the only purpose of doing so to make
controller log simpler?

And certainly, I have added Todd's comment in the wiki.

Thanks,
Dong


On Tue, Feb 28, 2017 at 1:37 PM, Jun Rao  wrote:

> Hi, Dong,
>
> 52. What you suggested would work. However, I am thinking that it's
> probably simpler to just set isNewReplica at the replica level. That way,
> the LeaderAndIsrRequest can be created a bit simpler. When reading a
> LeaderAndIsrRequest in the controller log, it's easier to see which
> replicas are new without looking at which broker the request is intended
> for.
>
> Could you also add those additional points from Todd's on 1 broker per disk
> vs JBOD vs RAID5/6 to the KIP?
>
> Thanks,
>
> Hi, Todd,
>
> Thanks for the feedback. That's very useful.
>
> Jun
>
> On Tue, Feb 28, 2017 at 10:25 AM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Certainly, I have added Todd to reply to the thread. And I have updated
> the
> > item to in the wiki.
> >
> > 50. The full statement is "Broker assumes a log directory to be good
> after
> > it starts, and mark log directory as bad once there is IOException when
> > broker attempts to access (i.e. read or write) the log directory". This
> > statement seems reasonable, right? If a log directory is actually bad,
> then
> > the broker will first assume it is OK, try to read logs on this log
> > directory, encounter IOException, and then mark it as bad.
> >
> > 51. My bad. I thought I removed it but I didn't. It is removed now.
> >
> > 52. I don't think so.. The isNewReplica field in the LeaderAndIsrRequest
> is
> > only relevant to the replica (i.e. broker) that receives the
> > LeaderAndIsrRequest. There is no need to specify whether each replica is
> > new inside LeaderAndIsrRequest. In other words, if a broker sends
> > LeaderAndIsrRequest to three different replicas of a given partition, the
> > isNewReplica field can be different across these three requests.
> >
> > Yeah, I would definitely want to start discussion on KIP-113 after we
> have
> > reached agreement on KIP-112. I have actually opened KIP-113 discussion
> > thread on 1/12 together with this thread. I have yet to add the ability
> to
> > list offline directories in KIP-113 which we discussed in this thread.
> >
> > Thanks for all your reviews! Is there further concern with the latest
> KIP?
> >
> > Thanks!
> > Dong
> >
> > On Tue, Feb 28, 2017 at 9:40 AM, Jun Rao  wrote:
> >
> > > Hi, Dong,
> > >
> > > RAID6 is an improvement over RAID5 and can tolerate 2 disks failure.
> > Eno's
> > > point is that the rebuild of RAID5/RAID6 requires reading more data
> > > compared with RAID10, which increases the probability of error during
> > > rebuild. This makes sense. In any case, do you think you could ask the
> > SREs
> > > at LinkedIn to share their opinions on RAID5/RAID6?
> > >
> > > Yes, when a replica is offline due to a bad disk, it makes sense to
> > handle
> > > it immediately as if a StopReplicaRequest is received (i.e., replica is
> > no
> > > longer considered a leader and is removed from any replica fetcher
> > thread).
> > > Could you add that detail in item 2. in the wiki?
> > >
> > > 50. The wiki says "Broker assumes a log directory to be good after it
> > > starts" : A log directory actually could be bad during startup.
> > >
> > > 51. In item 4, the wiki says "The controller watches the path
> > > /log_dir_event_notification for new znode.". This doesn't seem be
> needed
> > > now?
> > >
> > > 52. The isNewReplica field in LeaderAndIsrRequest should be for each
> > > replica inside the replicas field, right?
> > >
> > > Other than those, the current KIP looks good to me. Do you want to
> start
> > a
> > > separate discussion thread on KIP-113? I do have some comments there.
> > >
> > > Thanks for working on this!
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 27, 2017 at 5:51 PM, Dong Lin  wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > In addition to the Eno's reference of why rebuild time with RAID-5 is
> > > more
> > > > expensive, another concern is that RAID-5 will fail if more than one
> > disk
> > > > fails. JBOD is still works with 1+ disk failure and has better
> > > performance
> > > > with one disk failure. These seems like good argument for using JBOD
> > > > instead of RAID-5.
> > > >
> > > > If a leader replica goes offline, the broker should first take all
> > > actions
> > > > (i.e. remove the partition from fetcher thread) as if it has received
> > > > StopReplicaRequest for this partition because the replica can no
> longer
> > > > work anyway. It will also respond with error to any ProduceReq