[jira] [Commented] (KAFKA-4593) Task migration during rebalance callback process could lead the obsoleted task's IllegalStateException

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982470#comment-15982470
 ] 

Damian Guy commented on KAFKA-4593:
---

Is this even possible? I.e, thread A would have the lock on the state dir for 
that task, so Thread B would not start restoring. I think It would just spin 
trying to get the lock.

> Task migration during rebalance callback process could lead the obsoleted 
> task's IllegalStateException
> --
>
> Key: KAFKA-4593
> URL: https://issues.apache.org/jira/browse/KAFKA-4593
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: infrastructure
>
> 1. Assume 2 running threads A and B, and one task t1 jut for simplicity.
> 2. First rebalance is triggered, task t1 is assigned to A (B has no assigned 
> task).
> 3. During the first rebalance callback, task t1's state store need to be 
> restored on thread A, and this is called in "restoreActiveState" of 
> "createStreamTask".
> 4. Not suppose thread A has a long GC causing it to stall, a second rebalance 
> then will be triggered and kicked A out of the group; B gets the task t1 and 
> did the same restoration process, after the process thread B continues to 
> process data and update the state store, while at the same time writes more 
> messages to the changelog (so its log end offset has incremented).
> 5. After a while A resumes from the long GC, not knowing it has actually be 
> kicked out of the group and task t1 is no longer owned to itself, it 
> continues the restoration process but then realize that the log end offset 
> has advanced. When this happens, we will see the following exception on 
> thread A:
> {code}
> java.lang.IllegalStateException: task XXX Log end offset of
> YYY-table_stream-changelog-ZZ should not change while
> restoring: old end offset .., current offset ..
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.restoreActiveState(ProcessorStateManager.java:248)
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:201)
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.register(ProcessorContextImpl.java:122)
> at
> org.apache.kafka.streams.state.internals.RocksDBWindowStore.init(RocksDBWindowStore.java:200)
> at
> org.apache.kafka.streams.state.internals.MeteredWindowStore.init(MeteredWindowStore.java:65)
> at
> org.apache.kafka.streams.state.internals.CachingWindowStore.init(CachingWindowStore.java:65)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:86)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:120)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:794)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1222)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1195)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:897)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:71)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:240)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:230)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:314)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:278)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:261)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1039)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1004)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:570)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:359)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Edoardo Comar]

Congratulations Rajini !!!
Well deserved
--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU



From:   Gwen Shapira 
To: dev@kafka.apache.org, Users , 
priv...@kafka.apache.org
Date:   24/04/2017 22:07
Subject:[ANNOUNCE] New committer: Rajini Sivaram



The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
are pleased to announce that she has accepted!

Rajini contributed 83 patches, 8 KIPs (all security and quota
improvements) and a significant number of reviews. She is also on the
conference committee for Kafka Summit, where she helped select content
for our community event. Through her contributions she's shown good
judgement, good coding skills, willingness to work with the community on
finding the best
solutions and very consistent follow through on her work.

Thank you for your contributions, Rajini! Looking forward to many more :)

Gwen, for the Apache Kafka PMC



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Re: [VOTE] KIP-114: KTable state stores and improved semantics

[Eno Thereska]

Hello,

This KIP is now approved. The votes were:

Binding +1s: Guozhang, Sriram, Jay
Non-binding +1s: Matthias, Bill, Damian.

Many thanks
Eno
 
> On Apr 22, 2017, at 4:29 PM, Jay Kreps  wrote:
> 
> +1 Very well thought out.
> 
> -Jay
> 
> On Fri, Apr 21, 2017 at 10:39 AM Eno Thereska 
> wrote:
> 
>> Hi there,
>> 
>> Unless there are more issues on the discuss thread, I'd like to start the
>> vote on KIP-114.
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114%3A+KTable+state+stores+and+improved+semantics
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-114:+KTable+state+stores+and+improved+semantics
>>> .
>> 
>> Thanks
>> Eno

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Mickael Maison]

Congratulation Rajini !
Great news

On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar  wrote:
> Congratulations Rajini !!!
> Well deserved
> --
> Edoardo Comar
> IBM MessageHub
> eco...@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
>
>
>
> From:   Gwen Shapira 
> To: dev@kafka.apache.org, Users ,
> priv...@kafka.apache.org
> Date:   24/04/2017 22:07
> Subject:[ANNOUNCE] New committer: Rajini Sivaram
>
>
>
> The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
> are pleased to announce that she has accepted!
>
> Rajini contributed 83 patches, 8 KIPs (all security and quota
> improvements) and a significant number of reviews. She is also on the
> conference committee for Kafka Summit, where she helped select content
> for our community event. Through her contributions she's shown good
> judgement, good coding skills, willingness to work with the community on
> finding the best
> solutions and very consistent follow through on her work.
>
> Thank you for your contributions, Rajini! Looking forward to many more :)
>
> Gwen, for the Apache Kafka PMC
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[jira] [Commented] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982598#comment-15982598
 ] 

huxi commented on KAFKA-4879:
-

[~guozhang] What about KafkaConsumer.pollOnce? Does running time of 
`updateFetchPositions` count in the total timeout?

> KafkaConsumer.position may hang forever when deleting a topic
> -
>
> Key: KAFKA-4879
> URL: https://issues.apache.org/jira/browse/KAFKA-4879
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.0
>Reporter: Shixiong Zhu
>Assignee: Armin Braun
>
> KafkaConsumer.position may hang forever when deleting a topic. The problem is 
> this line 
> https://github.com/apache/kafka/blob/022bf129518e33e165f9ceefc4ab9e622952d3bd/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L374
> The timeout is "Long.MAX_VALUE", and it will just retry forever for 
> UnknownTopicOrPartitionException.
> Here is a reproducer
> {code}
> import org.apache.kafka.clients.consumer.KafkaConsumer;
> import org.apache.kafka.common.TopicPartition;
> import org.apache.kafka.common.serialization.StringDeserializer;
> import java.util.Collections;
> import java.util.Properties;
> import java.util.Set;
> public class KafkaReproducer {
>   public static void main(String[] args) {
> // Make sure "delete.topic.enable" is set to true.
> // Please create the topic test with "3" partitions manually.
> // The issue is gone when there is only one partition.
> String topic = "test";
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "testgroup");
> props.put("value.deserializer", StringDeserializer.class.getName());
> props.put("key.deserializer", StringDeserializer.class.getName());
> props.put("enable.auto.commit", "false");
> KafkaConsumer kc = new KafkaConsumer(props);
> kc.subscribe(Collections.singletonList(topic));
> kc.poll(0);
> Set partitions = kc.assignment();
> System.out.println("partitions: " + partitions);
> kc.pause(partitions);
> kc.seekToEnd(partitions);
> System.out.println("please delete the topic in 30 seconds");
> try {
>   // Sleep 30 seconds to give us enough time to delete the topic.
>   Thread.sleep(3);
> } catch (InterruptedException e) {
>   e.printStackTrace();
> }
> System.out.println("sleep end");
> for (TopicPartition p : partitions) {
>   System.out.println(p + " offset: " + kc.position(p));
> }
> System.out.println("cannot reach here");
> kc.close();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5122) Kafka Streams off-heap memory leak

[Jon Buffington (JIRA)]

Jon Buffington created KAFKA-5122:
-

 Summary: Kafka Streams off-heap memory leak
 Key: KAFKA-5122
 URL: https://issues.apache.org/jira/browse/KAFKA-5122
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.0
 Environment: Linux 64-bit
Oracle JVM version "1.8.0_121"
Reporter: Jon Buffington


I have a Kafka Streams application that leaks off-heap memory at a rate of 20MB 
per commit interval. The application is configured with a 1G heap; the heap 
memory does not show signs of leaking. The application reaches 16g of system 
memory usage before terminating and restarting.

Application facts:
* The data pipeline is source -> map -> groupByKey -> reduce -> to.
* The reduce operation uses a tumbling time window 
TimeWindows.of(TimeUnit.HOURS.toMillis(1)).until(TimeUnit.HOURS.toMillis(168)).
* The commit interval is five minutes (30ms).
* The application links to v0.10.2.0-cp1 of the Kakfa libraries. When I link to 
the current 0.10.2.1 RC3, the leak rate changes to ~10MB per commit interval.
* The application uses the schema registry for two pairs of serdes. One serde 
pair is used to read from a source topic that has 40 partitions. The other 
serde pair is used by the internal changelog and repartition topics created by 
the groupByKey/reduce operations.
* The source input rate varies between 500-1500 records/sec. The source rate 
variation does not change the size or frequency of the leak.
* The application heap has been configured using both 1024m and 2048m. The only 
observed difference between the two JVM heap sizes is more old gen collections 
at 1024m although there is little difference in throughput. JVM settings are 
{-server -Djava.awt.headless=true -Xss256k -XX:MaxMetaspaceSize=128m 
-XX:ReservedCodeCacheSize=64m -XX:CompressedClassSpaceSize=32m 
-XX:MaxDirectMemorySize=128m -XX:+AlwaysPreTouch -XX:+UseG1GC 
-XX:MaxGCPauseMillis=50 -XX:InitiatingHeapOccupancyPercent=35 
-XX:+PerfDisableSharedMem -XX:+UseStringDeduplication 
-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80}
* We configure a custom RocksDBConfigSetter to set 
options.setMaxBackgroundCompactions(Runtime.getRuntime.availableProcessors)
* Per 
,
 the SSTables are being compacted. Total disk usage for the state files 
(RocksDB) is ~2.5g. Per partition and window, there are 3-4 SSTables.
* The application is written in Scala and compiled using version 2.12.1.
• Oracle JVM version "1.8.0_121"

Various experiments that had no effect on the leak rate:
* Tried different RocksDB block sizes (4k, 16k, and 32k).
* Different numbers of instances (1, 2, and 4).
* Different numbers of threads (1, 4, 10, 40).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Damian Guy]

Congrats
On Tue, 25 Apr 2017 at 09:57, Mickael Maison 
wrote:

> Congratulation Rajini !
> Great news
>
> On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar  wrote:
> > Congratulations Rajini !!!
> > Well deserved
> > --
> > Edoardo Comar
> > IBM MessageHub
> > eco...@uk.ibm.com
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> > IBM United Kingdom Limited Registered in England and Wales with number
> > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > 3AU
> >
> >
> >
> > From:   Gwen Shapira 
> > To: dev@kafka.apache.org, Users ,
> > priv...@kafka.apache.org
> > Date:   24/04/2017 22:07
> > Subject:[ANNOUNCE] New committer: Rajini Sivaram
> >
> >
> >
> > The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
> > are pleased to announce that she has accepted!
> >
> > Rajini contributed 83 patches, 8 KIPs (all security and quota
> > improvements) and a significant number of reviews. She is also on the
> > conference committee for Kafka Summit, where she helped select content
> > for our community event. Through her contributions she's shown good
> > judgement, good coding skills, willingness to work with the community on
> > finding the best
> > solutions and very consistent follow through on her work.
> >
> > Thank you for your contributions, Rajini! Looking forward to many more :)
> >
> > Gwen, for the Apache Kafka PMC
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
>

[GitHub] kafka pull request #2911: MINOR: Improve information in assert failure for t...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2911

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown

This test is failing consistently in 
https://jenkins.confluent.io/job/kafka-trunk/,
but nowhere else. I ran this branch in a clone of that job several times 
and this
test didn't fail. I suggest we merge this PR, which improves the test, to 
help us
gather more information about the test failure.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
socket-server-test-metric-collection-after-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2911


commit aba3ed482e41db011974a44de72e57bc2e3b1a7d
Author: Ismael Juma 
Date:   2017-04-24T22:12:18Z

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

[Mickael Maison]

Even though it's a really small KIP, I'm sure people have ideas how to
improved it.

If there are no comments, I'll start a vote next week

On Thu, Mar 30, 2017 at 5:39 PM, Mickael Maison
 wrote:
> Hi all,
>
> We created KIP-137: Enhance TopicCommand --describe to show topics
> marked for deletion
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-137%3A+Enhance+TopicCommand+--describe+to+show+topics+marked+for+deletion
>
> Please help review the KIP. You feedback is appreciated!
>
> Thanks

[jira] [Commented] (KAFKA-5070) org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock the state directory: /opt/rocksdb/pulse10/0_18

[Dhana (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982716#comment-15982716
 ] 

Dhana commented on KAFKA-5070:
--

Hi Matthias J. Sax,

Thanks for reponse. We tried the 0.10.2.1, 0.10.2.1.rc2 version with this 2841 
fix reflected. We still get this error:
 at 
org.apache.kafka.streams.processor.internals.ProcessorStateManager.(ProcessorStateManager.java:100)

and

ProcessorStateException: Error opening store fmdbt at location 
/opt/rocksdb/pulse08/0_211/rocksdb/fmdbt.

> org.apache.kafka.streams.errors.LockException: task [0_18] Failed to lock the 
> state directory: /opt/rocksdb/pulse10/0_18
> 
>
> Key: KAFKA-5070
> URL: https://issues.apache.org/jira/browse/KAFKA-5070
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
> Environment: Linux Version
>Reporter: Dhana
> Attachments: RocksDB_LockStateDirec.7z
>
>
> Notes: we run two instance of consumer in two difference machines/nodes.
> we have 400 partitions. 200  stream threads/consumer, with 2 consumer.
> We perform HA test(on rebalance - shutdown of one of the consumer/broker), we 
> see this happening
> Error:
> 2017-04-05 11:36:09.352 WARN  StreamThread:1184 StreamThread-66 - Could not 
> create task 0_115. Will retry.
> org.apache.kafka.streams.errors.LockException: task [0_115] Failed to lock 
> the state directory: /opt/rocksdb/pulse10/0_115
>   at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.(ProcessorStateManager.java:102)
>   at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:73)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:108)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

[Ismael Juma]

Thanks for the KIP. Would it make sense for MarkedForDeletion to be before
`Configs`? I can see arguments both ways, so I was wondering what your
thoughts were?

Ismael

On Thu, Mar 30, 2017 at 5:39 PM, Mickael Maison 
wrote:

> Hi all,
>
> We created KIP-137: Enhance TopicCommand --describe to show topics
> marked for deletion
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 137%3A+Enhance+TopicCommand+--describe+to+show+topics+marked+for+deletion
>
> Please help review the KIP. You feedback is appreciated!
>
> Thanks
>

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Rajini Sivaram]

Thanks everyone!

It has been a pleasure working with all of you in the Kafka community. Many
thanks to the PMC for this exciting opportunity.

Regards,

Rajini

On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy  wrote:

> Congrats
> On Tue, 25 Apr 2017 at 09:57, Mickael Maison 
> wrote:
>
> > Congratulation Rajini !
> > Great news
> >
> > On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar 
> wrote:
> > > Congratulations Rajini !!!
> > > Well deserved
> > > --
> > > Edoardo Comar
> > > IBM MessageHub
> > > eco...@uk.ibm.com
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > > IBM United Kingdom Limited Registered in England and Wales with number
> > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> > PO6
> > > 3AU
> > >
> > >
> > >
> > > From:   Gwen Shapira 
> > > To: dev@kafka.apache.org, Users ,
> > > priv...@kafka.apache.org
> > > Date:   24/04/2017 22:07
> > > Subject:[ANNOUNCE] New committer: Rajini Sivaram
> > >
> > >
> > >
> > > The PMC for Apache Kafka has invited Rajini Sivaram as a committer and
> we
> > > are pleased to announce that she has accepted!
> > >
> > > Rajini contributed 83 patches, 8 KIPs (all security and quota
> > > improvements) and a significant number of reviews. She is also on the
> > > conference committee for Kafka Summit, where she helped select content
> > > for our community event. Through her contributions she's shown good
> > > judgement, good coding skills, willingness to work with the community
> on
> > > finding the best
> > > solutions and very consistent follow through on her work.
> > >
> > > Thank you for your contributions, Rajini! Looking forward to many more
> :)
> > >
> > > Gwen, for the Apache Kafka PMC
> > >
> > >
> > >
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> >
>

Re: [DISCUSS] KIP-136: Add Listener name and Security Protocol name to SelectorMetrics tags

[Edoardo Comar]

This is another very small KIP.

We've implemented it in our clusters' and found it useful to look at our 
metrics.
More comments welcome; I'd like to do the voting next week.

Edo
--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU



From:   Roger Hoover 
To: "dev@kafka.apache.org" 
Date:   30/03/2017 17:53
Subject:Re: [DISCUSS] KIP-136: Add Listener name and Security 
Protocol name to SelectorMetrics tags



Edo,

Thanks for the proposal.  This looks great to me.

Cheers,

Roger

On Thu, Mar 30, 2017 at 8:51 AM, Edoardo Comar  wrote:

> Hi all,
>
> We created KIP-136: Add Listener name and Security Protocol name to
> SelectorMetrics tags
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 136%3A+Add+Listener+name+and+Security+Protocol+name+to+
> SelectorMetrics+tags
>
> Please help review the KIP. You feedback is appreciated!
>
> cheers,
> Edo
> --
> Edoardo Comar
> IBM MessageHub
> eco...@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. 
PO6
> 3AU
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 
3AU
>



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Eno Thereska]

Congrats!

Eno
> On Apr 25, 2017, at 12:17 PM, Rajini Sivaram  wrote:
> 
> Thanks everyone!
> 
> It has been a pleasure working with all of you in the Kafka community. Many
> thanks to the PMC for this exciting opportunity.
> 
> Regards,
> 
> Rajini
> 
> On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy  wrote:
> 
>> Congrats
>> On Tue, 25 Apr 2017 at 09:57, Mickael Maison 
>> wrote:
>> 
>>> Congratulation Rajini !
>>> Great news
>>> 
>>> On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar 
>> wrote:
 Congratulations Rajini !!!
 Well deserved
 --
 Edoardo Comar
 IBM MessageHub
 eco...@uk.ibm.com
 IBM UK Ltd, Hursley Park, SO21 2JN
 
 IBM United Kingdom Limited Registered in England and Wales with number
 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
>>> PO6
 3AU
 
 
 
 From:   Gwen Shapira 
 To: dev@kafka.apache.org, Users ,
 priv...@kafka.apache.org
 Date:   24/04/2017 22:07
 Subject:[ANNOUNCE] New committer: Rajini Sivaram
 
 
 
 The PMC for Apache Kafka has invited Rajini Sivaram as a committer and
>> we
 are pleased to announce that she has accepted!
 
 Rajini contributed 83 patches, 8 KIPs (all security and quota
 improvements) and a significant number of reviews. She is also on the
 conference committee for Kafka Summit, where she helped select content
 for our community event. Through her contributions she's shown good
 judgement, good coding skills, willingness to work with the community
>> on
 finding the best
 solutions and very consistent follow through on her work.
 
 Thank you for your contributions, Rajini! Looking forward to many more
>> :)
 
 Gwen, for the Apache Kafka PMC
 
 
 
 Unless stated otherwise above:
 IBM United Kingdom Limited - Registered in England and Wales with
>> number
 741598.
 Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>>> 3AU
>>> 
>> 

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Molnár Bálint]

Congrats, Rajini:)

2017-04-25 13:31 GMT+02:00 Eno Thereska :

> Congrats!
>
> Eno
> > On Apr 25, 2017, at 12:17 PM, Rajini Sivaram 
> wrote:
> >
> > Thanks everyone!
> >
> > It has been a pleasure working with all of you in the Kafka community.
> Many
> > thanks to the PMC for this exciting opportunity.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy 
> wrote:
> >
> >> Congrats
> >> On Tue, 25 Apr 2017 at 09:57, Mickael Maison 
> >> wrote:
> >>
> >>> Congratulation Rajini !
> >>> Great news
> >>>
> >>> On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar 
> >> wrote:
>  Congratulations Rajini !!!
>  Well deserved
>  --
>  Edoardo Comar
>  IBM MessageHub
>  eco...@uk.ibm.com
>  IBM UK Ltd, Hursley Park, SO21 2JN
> 
>  IBM United Kingdom Limited Registered in England and Wales with number
>  741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> >>> PO6
>  3AU
> 
> 
> 
>  From:   Gwen Shapira 
>  To: dev@kafka.apache.org, Users ,
>  priv...@kafka.apache.org
>  Date:   24/04/2017 22:07
>  Subject:[ANNOUNCE] New committer: Rajini Sivaram
> 
> 
> 
>  The PMC for Apache Kafka has invited Rajini Sivaram as a committer and
> >> we
>  are pleased to announce that she has accepted!
> 
>  Rajini contributed 83 patches, 8 KIPs (all security and quota
>  improvements) and a significant number of reviews. She is also on the
>  conference committee for Kafka Summit, where she helped select content
>  for our community event. Through her contributions she's shown good
>  judgement, good coding skills, willingness to work with the community
> >> on
>  finding the best
>  solutions and very consistent follow through on her work.
> 
>  Thank you for your contributions, Rajini! Looking forward to many more
> >> :)
> 
>  Gwen, for the Apache Kafka PMC
> 
> 
> 
>  Unless stated otherwise above:
>  IBM United Kingdom Limited - Registered in England and Wales with
> >> number
>  741598.
>  Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> >>> 3AU
> >>>
> >>
>
>

Re: [DISCUSS] KIP-136: Add Listener name and Security Protocol name to SelectorMetrics tags

[Ismael Juma]

Thanks for the KIP. I think it makes sense to have those tags. My only
question is regarding the compatibility impact. We don't have a good
compatibility story when it comes to adding tags to existing metrics since
the JmxReporter adds the tags to the object name.

Ismael

On Thu, Mar 30, 2017 at 4:51 PM, Edoardo Comar  wrote:

> Hi all,
>
> We created KIP-136: Add Listener name and Security Protocol name to
> SelectorMetrics tags
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 136%3A+Add+Listener+name+and+Security+Protocol+name+to+
> SelectorMetrics+tags
>
> Please help review the KIP. You feedback is appreciated!
>
> cheers,
> Edo
> --
> Edoardo Comar
> IBM MessageHub
> eco...@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

[jira] [Assigned] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun reassigned KAFKA-4879:
--

Assignee: (was: Armin Braun)

> KafkaConsumer.position may hang forever when deleting a topic
> -
>
> Key: KAFKA-4879
> URL: https://issues.apache.org/jira/browse/KAFKA-4879
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.0
>Reporter: Shixiong Zhu
>
> KafkaConsumer.position may hang forever when deleting a topic. The problem is 
> this line 
> https://github.com/apache/kafka/blob/022bf129518e33e165f9ceefc4ab9e622952d3bd/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L374
> The timeout is "Long.MAX_VALUE", and it will just retry forever for 
> UnknownTopicOrPartitionException.
> Here is a reproducer
> {code}
> import org.apache.kafka.clients.consumer.KafkaConsumer;
> import org.apache.kafka.common.TopicPartition;
> import org.apache.kafka.common.serialization.StringDeserializer;
> import java.util.Collections;
> import java.util.Properties;
> import java.util.Set;
> public class KafkaReproducer {
>   public static void main(String[] args) {
> // Make sure "delete.topic.enable" is set to true.
> // Please create the topic test with "3" partitions manually.
> // The issue is gone when there is only one partition.
> String topic = "test";
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "testgroup");
> props.put("value.deserializer", StringDeserializer.class.getName());
> props.put("key.deserializer", StringDeserializer.class.getName());
> props.put("enable.auto.commit", "false");
> KafkaConsumer kc = new KafkaConsumer(props);
> kc.subscribe(Collections.singletonList(topic));
> kc.poll(0);
> Set partitions = kc.assignment();
> System.out.println("partitions: " + partitions);
> kc.pause(partitions);
> kc.seekToEnd(partitions);
> System.out.println("please delete the topic in 30 seconds");
> try {
>   // Sleep 30 seconds to give us enough time to delete the topic.
>   Thread.sleep(3);
> } catch (InterruptedException e) {
>   e.printStackTrace();
> }
> System.out.println("sleep end");
> for (TopicPartition p : partitions) {
>   System.out.println(p + " offset: " + kc.position(p));
> }
> System.out.println("cannot reach here");
> kc.close();
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5123) Refactor ZkUtils readData* methods

[Balint Molnar (JIRA)]

Balint Molnar created KAFKA-5123:


 Summary: Refactor ZkUtils readData* methods 
 Key: KAFKA-5123
 URL: https://issues.apache.org/jira/browse/KAFKA-5123
 Project: Kafka
  Issue Type: Bug
Reporter: Balint Molnar
Assignee: Balint Molnar
Priority: Minor


Usually only the data value is required but every readData method in the 
ZkUtils returns a Tuple with the data and the stat.

https://github.com/apache/kafka/pull/2888



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5123) Refactor ZkUtils readData* methods

[Balint Molnar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5123 started by Balint Molnar.

> Refactor ZkUtils readData* methods 
> ---
>
> Key: KAFKA-5123
> URL: https://issues.apache.org/jira/browse/KAFKA-5123
> Project: Kafka
>  Issue Type: Bug
>Reporter: Balint Molnar
>Assignee: Balint Molnar
>Priority: Minor
>
> Usually only the data value is required but every readData method in the 
> ZkUtils returns a Tuple with the data and the stat.
> https://github.com/apache/kafka/pull/2888



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5025) FetchRequestTest should use batches with more than one message

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5025 started by Armin Braun.
--
> FetchRequestTest should use batches with more than one message
> --
>
> Key: KAFKA-5025
> URL: https://issues.apache.org/jira/browse/KAFKA-5025
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> As part of the message format changes for KIP-98, 
> FetchRequestTest.produceData was changed to always use record batches 
> containing a single message. We should restructure the test so that it's more 
> realistic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-4928) Add integration test for DumpLogSegments

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4928 started by Armin Braun.
--
> Add integration test for DumpLogSegments
> 
>
> Key: KAFKA-4928
> URL: https://issues.apache.org/jira/browse/KAFKA-4928
> Project: Kafka
>  Issue Type: Test
>Reporter: Ismael Juma
>Assignee: Armin Braun
>  Labels: newbie
> Fix For: 0.11.0.0
>
>
> DumpLogSegments is an important tool to analyse log files, but we have no 
> JUnit tests for it. It would be good to have some tests that verify that the 
> output is sane for a populated log.
> Our system tests call DumpLogSegments, but we should be able to detect 
> regressions via the JUnit test suite.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5018) LogCleaner tests to verify behaviour of message format v2

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun updated KAFKA-5018:
---
Status: Patch Available  (was: Open)

> LogCleaner tests to verify behaviour of message format v2
> -
>
> Key: KAFKA-5018
> URL: https://issues.apache.org/jira/browse/KAFKA-5018
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> It would be good to add LogCleaner tests to verify the behaviour of fields 
> like baseOffset after compaction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4928) Add integration test for DumpLogSegments

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun updated KAFKA-4928:
---
Status: Patch Available  (was: In Progress)

> Add integration test for DumpLogSegments
> 
>
> Key: KAFKA-4928
> URL: https://issues.apache.org/jira/browse/KAFKA-4928
> Project: Kafka
>  Issue Type: Test
>Reporter: Ismael Juma
>Assignee: Armin Braun
>  Labels: newbie
> Fix For: 0.11.0.0
>
>
> DumpLogSegments is an important tool to analyse log files, but we have no 
> JUnit tests for it. It would be good to have some tests that verify that the 
> output is sane for a populated log.
> Our system tests call DumpLogSegments, but we should be able to detect 
> regressions via the JUnit test suite.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work stopped] (KAFKA-5025) FetchRequestTest should use batches with more than one message

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5025 stopped by Armin Braun.
--
> FetchRequestTest should use batches with more than one message
> --
>
> Key: KAFKA-5025
> URL: https://issues.apache.org/jira/browse/KAFKA-5025
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> As part of the message format changes for KIP-98, 
> FetchRequestTest.produceData was changed to always use record batches 
> containing a single message. We should restructure the test so that it's more 
> realistic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5025) FetchRequestTest should use batches with more than one message

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun reassigned KAFKA-5025:
--

Assignee: (was: Armin Braun)

> FetchRequestTest should use batches with more than one message
> --
>
> Key: KAFKA-5025
> URL: https://issues.apache.org/jira/browse/KAFKA-5025
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
> Fix For: 0.11.0.0
>
>
> As part of the message format changes for KIP-98, 
> FetchRequestTest.produceData was changed to always use record batches 
> containing a single message. We should restructure the test so that it's more 
> realistic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Eno Thereska (JIRA)]

Eno Thereska created KAFKA-5124:
---

 Summary: shouldInnerLeftJoin unit test fails
 Key: KAFKA-5124
 URL: https://issues.apache.org/jira/browse/KAFKA-5124
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.0
Reporter: Eno Thereska
 Fix For: 0.11.0.0


Unit test on trunk gives occasional failure:
org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin FAILED
java.lang.AssertionError: Condition not met within timeout 3. Expecting 
1 records from topic output- while only received 0: []
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
at 
org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
at 
org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982909#comment-15982909
 ] 

Armin Braun commented on KAFKA-5124:


Trying this one, I can reproduce it locally :)

> shouldInnerLeftJoin unit test fails
> ---
>
> Key: KAFKA-5124
> URL: https://issues.apache.org/jira/browse/KAFKA-5124
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> Unit test on trunk gives occasional failure:
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
> shouldInnerLeftJoin FAILED
> java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 1 records from topic output- while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun reassigned KAFKA-5124:
--

Assignee: Armin Braun

> shouldInnerLeftJoin unit test fails
> ---
>
> Key: KAFKA-5124
> URL: https://issues.apache.org/jira/browse/KAFKA-5124
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> Unit test on trunk gives occasional failure:
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
> shouldInnerLeftJoin FAILED
> java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 1 records from topic output- while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Armin Braun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Armin Braun updated KAFKA-5124:
---
Issue Type: Sub-task  (was: Bug)
Parent: KAFKA-2054

> shouldInnerLeftJoin unit test fails
> ---
>
> Key: KAFKA-5124
> URL: https://issues.apache.org/jira/browse/KAFKA-5124
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> Unit test on trunk gives occasional failure:
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
> shouldInnerLeftJoin FAILED
> java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 1 records from topic output- while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982917#comment-15982917
 ] 

Eno Thereska commented on KAFKA-5124:
-

Thanks [~original-brownbear]

> shouldInnerLeftJoin unit test fails
> ---
>
> Key: KAFKA-5124
> URL: https://issues.apache.org/jira/browse/KAFKA-5124
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> Unit test on trunk gives occasional failure:
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
> shouldInnerLeftJoin FAILED
> java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 1 records from topic output- while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5025) FetchRequestTest should use batches with more than one message

[Umesh Chaudhary (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Chaudhary reassigned KAFKA-5025:
--

Assignee: Umesh Chaudhary

> FetchRequestTest should use batches with more than one message
> --
>
> Key: KAFKA-5025
> URL: https://issues.apache.org/jira/browse/KAFKA-5025
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Umesh Chaudhary
> Fix For: 0.11.0.0
>
>
> As part of the message format changes for KIP-98, 
> FetchRequestTest.produceData was changed to always use record batches 
> containing a single message. We should restructure the test so that it's more 
> realistic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5025) FetchRequestTest should use batches with more than one message

[Umesh Chaudhary (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982953#comment-15982953
 ] 

Umesh Chaudhary commented on KAFKA-5025:


[~ijuma], taking this one to start my contribution to the project. May I ask 
some guidelines to start working on this ?

> FetchRequestTest should use batches with more than one message
> --
>
> Key: KAFKA-5025
> URL: https://issues.apache.org/jira/browse/KAFKA-5025
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Umesh Chaudhary
> Fix For: 0.11.0.0
>
>
> As part of the message format changes for KIP-98, 
> FetchRequestTest.produceData was changed to always use record batches 
> containing a single message. We should restructure the test so that it's more 
> realistic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2858: KAFKA-438: Code cleanup in MessageTest

[jozi-k]

Github user jozi-k closed the pull request at:

https://github.com/apache/kafka/pull/2858


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-438) Code cleanup in MessageTest

[Jozef Koval (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982964#comment-15982964
 ] 

Jozef Koval commented on KAFKA-438:
---

Can be closed as MessageTest will be deleted soon.

> Code cleanup in MessageTest
> ---
>
> Key: KAFKA-438
> URL: https://issues.apache.org/jira/browse/KAFKA-438
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.7.1
>Reporter: Jim Plush
>Priority: Trivial
> Attachments: KAFKA-438
>
>
> While exploring the Unit Tests this class had an unused import statement, 
> some ambiguity on which HashMap implementation was being used and assignments 
> of function returns when not required. 
> Trivial stuff



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-438) Code cleanup in MessageTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982961#comment-15982961
 ] 

ASF GitHub Bot commented on KAFKA-438:
--

Github user jozi-k closed the pull request at:

https://github.com/apache/kafka/pull/2858


> Code cleanup in MessageTest
> ---
>
> Key: KAFKA-438
> URL: https://issues.apache.org/jira/browse/KAFKA-438
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.7.1
>Reporter: Jim Plush
>Priority: Trivial
> Attachments: KAFKA-438
>
>
> While exploring the Unit Tests this class had an unused import statement, 
> some ambiguity on which HashMap implementation was being used and assignments 
> of function returns when not required. 
> Trivial stuff



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5120) Several controller metrics block if controller lock is held by another thread

[Tim Carey-Smith (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Carey-Smith updated KAFKA-5120:
---
Description: 
We have been tracking latency issues surrounding queries to Controller MBeans. 
Upon digging into the root causes, we discovered that several metrics acquire 
the controller lock within the gauge. 

The affected metrics are: 

* {{ActiveControllerCount}}
* {{OfflinePartitionsCount}}
* {{PreferredReplicaImbalanceCount}}

If the controller is currently holding the lock and a MBean request is 
received, the thread executing the request will block until the controller 
releases the lock. 

We discovered this in a cluster where the controller was holding the lock for 
extended periods of time for normal operations. We have documented this issue 
in KAFKA-5116. 

Several possible solutions exist: 

* Remove the lock from inside these {{Gauge}} s. 
* Store and update the metric values in {{AtomicLong}} s. 

Modifying the {{ActiveControllerCount}} metric seems to be straight-forward 
while the other 2 metrics seem to be more involved. 

We're happy to contribute a patch, but wanted to discuss potential solutions 
and their tradeoffs before proceeding. 

  was:
We have been tracking latency issues surrounding queries to Controller MBeans. 
Upon digging into the root causes, we discovered that several metrics acquire 
the controller lock within the gauge. 

The affected metrics are: 

* {{ActiveControllerCount}}
* {{OfflinePartitionsCount}}
* {{PreferredReplicaImbalanceCount}}

If the controller is currently holding the lock and a MBean request is 
received, the thread executing the request will block until the controller 
releases the lock. 

We discovered this in a cluster where the controller was holding the lock for 
extended periods of time for normal operations. We have documented this issue 
in KAFKA-5116. 

Several possible solutions exist: 

* Remove the lock from inside these {{Gauge}}s. 
* Store and update the metric values in {{AtomicLong}}s. 

Modifying the {{ActiveControllerCount}} metric seems to be straight-forward 
while the other 2 metrics seem to be more involved. 

We're happy to contribute a patch, but wanted to discuss potential solutions 
and their tradeoffs before proceeding. 


> Several controller metrics block if controller lock is held by another thread
> -
>
> Key: KAFKA-5120
> URL: https://issues.apache.org/jira/browse/KAFKA-5120
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, metrics
>Affects Versions: 0.10.2.0
>Reporter: Tim Carey-Smith
>Priority: Minor
>
> We have been tracking latency issues surrounding queries to Controller 
> MBeans. Upon digging into the root causes, we discovered that several metrics 
> acquire the controller lock within the gauge. 
> The affected metrics are: 
> * {{ActiveControllerCount}}
> * {{OfflinePartitionsCount}}
> * {{PreferredReplicaImbalanceCount}}
> If the controller is currently holding the lock and a MBean request is 
> received, the thread executing the request will block until the controller 
> releases the lock. 
> We discovered this in a cluster where the controller was holding the lock for 
> extended periods of time for normal operations. We have documented this issue 
> in KAFKA-5116. 
> Several possible solutions exist: 
> * Remove the lock from inside these {{Gauge}} s. 
> * Store and update the metric values in {{AtomicLong}} s. 
> Modifying the {{ActiveControllerCount}} metric seems to be straight-forward 
> while the other 2 metrics seem to be more involved. 
> We're happy to contribute a patch, but wanted to discuss potential solutions 
> and their tradeoffs before proceeding. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5120) Several controller metrics block if controller lock is held by another thread

[Tim Carey-Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982994#comment-15982994
 ] 

Tim Carey-Smith commented on KAFKA-5120:


[~onurkaraman] Ah, thanks! That patch looks amazing. What is the timeframe for 
that patch landing? Having safe access to these metrics will be excellent. 

> Several controller metrics block if controller lock is held by another thread
> -
>
> Key: KAFKA-5120
> URL: https://issues.apache.org/jira/browse/KAFKA-5120
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, metrics
>Affects Versions: 0.10.2.0
>Reporter: Tim Carey-Smith
>Priority: Minor
>
> We have been tracking latency issues surrounding queries to Controller 
> MBeans. Upon digging into the root causes, we discovered that several metrics 
> acquire the controller lock within the gauge. 
> The affected metrics are: 
> * {{ActiveControllerCount}}
> * {{OfflinePartitionsCount}}
> * {{PreferredReplicaImbalanceCount}}
> If the controller is currently holding the lock and a MBean request is 
> received, the thread executing the request will block until the controller 
> releases the lock. 
> We discovered this in a cluster where the controller was holding the lock for 
> extended periods of time for normal operations. We have documented this issue 
> in KAFKA-5116. 
> Several possible solutions exist: 
> * Remove the lock from inside these {{Gauge}} s. 
> * Store and update the metric values in {{AtomicLong}} s. 
> Modifying the {{ActiveControllerCount}} metric seems to be straight-forward 
> while the other 2 metrics seem to be more involved. 
> We're happy to contribute a patch, but wanted to discuss potential solutions 
> and their tradeoffs before proceeding. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5120) Several controller metrics block if controller lock is held by another thread

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983009#comment-15983009
 ] 

Ismael Juma commented on KAFKA-5120:


[~halorgium], if everything goes well, that PR should be merged this week.

> Several controller metrics block if controller lock is held by another thread
> -
>
> Key: KAFKA-5120
> URL: https://issues.apache.org/jira/browse/KAFKA-5120
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, metrics
>Affects Versions: 0.10.2.0
>Reporter: Tim Carey-Smith
>Priority: Minor
>
> We have been tracking latency issues surrounding queries to Controller 
> MBeans. Upon digging into the root causes, we discovered that several metrics 
> acquire the controller lock within the gauge. 
> The affected metrics are: 
> * {{ActiveControllerCount}}
> * {{OfflinePartitionsCount}}
> * {{PreferredReplicaImbalanceCount}}
> If the controller is currently holding the lock and a MBean request is 
> received, the thread executing the request will block until the controller 
> releases the lock. 
> We discovered this in a cluster where the controller was holding the lock for 
> extended periods of time for normal operations. We have documented this issue 
> in KAFKA-5116. 
> Several possible solutions exist: 
> * Remove the lock from inside these {{Gauge}} s. 
> * Store and update the metric values in {{AtomicLong}} s. 
> Modifying the {{ActiveControllerCount}} metric seems to be straight-forward 
> while the other 2 metrics seem to be more involved. 
> We're happy to contribute a patch, but wanted to discuss potential solutions 
> and their tradeoffs before proceeding. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2912: KAFKA-4942 Fix commitTimeoutMs being set before th...

[56quarters]

GitHub user 56quarters opened a pull request:

https://github.com/apache/kafka/pull/2912

KAFKA-4942 Fix commitTimeoutMs being set before the commit actually started

This fixes KAFKA-4942

This supersededs #2730 

/cc @simplesteph @gwenshap @ewencp

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smarter-travel-media/kafka 
fix-connect-offset-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2912


commit f93bd001a723c5e1402bf92474f43fb0b991a44c
Author: simplesteph 
Date:   2017-03-24T00:03:07Z

Fixed commitTimeoutMs being set before the commit actually started

Fixes KAFKA-4942

commit e7b704d97b8de35384f6d24ba48f050a0b20be01
Author: Nick Pillitteri 
Date:   2017-04-21T19:49:37Z

Test for commitTimeoutMs being set before commit started

See KAFKA-4942




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4942) Kafka Connect: Offset committing times out before expected

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983038#comment-15983038
 ] 

ASF GitHub Bot commented on KAFKA-4942:
---

GitHub user 56quarters opened a pull request:

https://github.com/apache/kafka/pull/2912

KAFKA-4942 Fix commitTimeoutMs being set before the commit actually started

This fixes KAFKA-4942

This supersededs #2730 

/cc @simplesteph @gwenshap @ewencp

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smarter-travel-media/kafka 
fix-connect-offset-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2912


commit f93bd001a723c5e1402bf92474f43fb0b991a44c
Author: simplesteph 
Date:   2017-03-24T00:03:07Z

Fixed commitTimeoutMs being set before the commit actually started

Fixes KAFKA-4942

commit e7b704d97b8de35384f6d24ba48f050a0b20be01
Author: Nick Pillitteri 
Date:   2017-04-21T19:49:37Z

Test for commitTimeoutMs being set before commit started

See KAFKA-4942




> Kafka Connect: Offset committing times out before expected
> --
>
> Key: KAFKA-4942
> URL: https://issues.apache.org/jira/browse/KAFKA-4942
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Stephane Maarek
>
> On Kafka 0.10.2.0
> I run a connector that deals with a lot of data, in a kafka connect cluster
> When the offsets are getting committed, I get the following:
> {code}
> [2017-03-23 03:56:25,134] INFO WorkerSinkTask{id=MyConnector-1} Committing 
> offsets (org.apache.kafka.connect.runtime.WorkerSinkTask)
> [2017-03-23 03:56:25,135] WARN Commit of WorkerSinkTask{id=MyConnector-1} 
> offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask)
> {code}
> If you look at the timestamps, they're 1 ms apart. My settings are the 
> following: 
> {code}
>   offset.flush.interval.ms = 12
>   offset.flush.timeout.ms = 6
>   offset.storage.topic = _connect_offsets
> {code}
> It seems the offset flush timeout setting is completely ignored for the look 
> of the logs. I would expect the timeout message to happen 60 seconds after 
> the commit offset INFO message, not 1 millisecond later.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4970) Add a 'ProducerIdResource' to enable authorization for generating producer ids

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983095#comment-15983095
 ] 

Ismael Juma commented on KAFKA-4970:


In the KIP we mention `ProducerTransactionalId` as a ResourceType. Is this the 
same as that or something else?

> Add a 'ProducerIdResource' to enable authorization for generating producer ids
> --
>
> Key: KAFKA-4970
> URL: https://issues.apache.org/jira/browse/KAFKA-4970
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> With the KIP-98 idempotent producer, we introduce a new `InitPidRequest` 
> which is called by a producer to get a producer id which is used to 
> deduplicate messages on the broker. The broker will allocate a new producer 
> id upon the receipt of this request and return it to the client. 
> Currently, there is no authorization on the producer Id space. It would be 
> good to add a producer id resource. This would mean that only authorized 
> clients would get pids, and only authorized clients would be able to send 
> messages which set the pid and sequence number.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

[Vahid S Hashemian]

Thanks for the KIP Mickael. 
Looks good. I also prefer 'MarkedForDeletion' before 'Configs'.

--Vahid



From:   Ismael Juma 
To: dev@kafka.apache.org
Date:   04/25/2017 04:15 AM
Subject:Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to 
show topics marked for deletion
Sent by:isma...@gmail.com



Thanks for the KIP. Would it make sense for MarkedForDeletion to be before
`Configs`? I can see arguments both ways, so I was wondering what your
thoughts were?

Ismael

On Thu, Mar 30, 2017 at 5:39 PM, Mickael Maison 
wrote:

> Hi all,
>
> We created KIP-137: Enhance TopicCommand --describe to show topics
> marked for deletion
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 
137%3A+Enhance+TopicCommand+--describe+to+show+topics+marked+for+deletion
>
> Please help review the KIP. You feedback is appreciated!
>
> Thanks
>

[GitHub] kafka pull request #2913: Kafka-4994: Fix findbug warnings about OffsetStora...

[johnma14]

GitHub user johnma14 opened a pull request:

https://github.com/apache/kafka/pull/2913

Kafka-4994: Fix findbug warnings about OffsetStorageWriter

OffsetStorageWriter is not a thread-safe class and should be accessed
only from a Task's processing thread. The WorkerSourceTask class calls
the different methods (offset, beginFlush, cancelFlush, handleFinishWrite)
within a synchronized block. Hence the method definitions in 
OffsetStorageWriter.java does not need to contain the keyword synchronized
again.

In the OffsetStorageWriter.java class, the doFlush() method is not 
explicitely
synchronized like the other methods in this class. Hence this can lead to
inconsistent synchronization of variables like currentFlushId and toFlush 
which
are set in the synchronized methods within this class.

- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageWriter.java
- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L295


Closes bug: Kafka-4994

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnma14/kafka bug/kafka-4994

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2913






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5125) controlled.shutdown.enable=true leads blocking some time during shutdown when the broker is the only remaining broker

[Zeynep Arikoglu (JIRA)]

Zeynep Arikoglu created KAFKA-5125:
--

 Summary: controlled.shutdown.enable=true leads blocking some time 
during shutdown when the broker is the only remaining broker
 Key: KAFKA-5125
 URL: https://issues.apache.org/jira/browse/KAFKA-5125
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.10.2.0, 0.10.1.1
 Environment: Ubuntu 16
Reporter: Zeynep Arikoglu
 Attachments: Stack dump.pdf

The documentation clearly states 'Note that controlled shutdown will only 
succeed if all the partitions hosted on the broker have replicas' but
even if we have replication, when you try to shutdown the broker that is 
serving as the only remaining broker for a partition, you will end up blocking 
for some time if the controlled shutdown is set to true. Say we have two 
brokers and our topics are replicated. When we shutdown the first one, it will 
terminate gracefully with controlled shutdown set to true. When we shutdown the 
second one, it will block and would have to be killed forcefully after a minute 
or so.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4970) Add a 'ProducerIdResource' to enable authorization for generating producer ids

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983276#comment-15983276
 ] 

Apurva Mehta commented on KAFKA-4970:
-

This is different. This is to make the  'producerId' itself a resource whether 
or not transactions are being used. The main motivation is to ensure that only 
authorized clients can set pids and sequence numbers in their messages, which 
is different from authorizing transactional clients, which is the only thing 
the KIP currently deals with.

> Add a 'ProducerIdResource' to enable authorization for generating producer ids
> --
>
> Key: KAFKA-4970
> URL: https://issues.apache.org/jira/browse/KAFKA-4970
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>
> With the KIP-98 idempotent producer, we introduce a new `InitPidRequest` 
> which is called by a producer to get a producer id which is used to 
> deduplicate messages on the broker. The broker will allocate a new producer 
> id upon the receipt of this request and return it to the client. 
> Currently, there is no authorization on the producer Id space. It would be 
> good to add a producer id resource. This would mean that only authorized 
> clients would get pids, and only authorized clients would be able to send 
> messages which set the pid and sequence number.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2659: KAFKA-4840 : BufferPool errors can cause buffer po...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2659


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4840) There are are still cases where producer buffer pool will not remove waiters.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983300#comment-15983300
 ] 

ASF GitHub Bot commented on KAFKA-4840:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2659


> There are are still cases where producer buffer pool will not remove waiters.
> -
>
> Key: KAFKA-4840
> URL: https://issues.apache.org/jira/browse/KAFKA-4840
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.0
>Reporter: Sean McCauliff
>
> There are several problems dealing with errors in  BufferPool.allocate(int 
> size, long maxTimeToBlockMs):
> * The accumulated number of bytes are not put back into the available pool 
> when an exception happens and a thread is waiting for bytes to become 
> available.  This will cause the capacity of the buffer pool to decrease over 
> time any time a timeout is hit within this method.
> * If a Throwable other than InterruptedException is thrown out of await() for 
> some reason or if there is an exception thrown in the corresponding finally 
> block around the await(), for example if waitTime.record(.) throws an 
> exception, then the waiters are not removed from the waiters deque.
> * On timeout or other exception waiters could be signaled, but are not.  If 
> no other buffers are freed then the next waiting thread will also timeout and 
> so on.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[VOTE] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Becket Qin]

Hi,

I would like to start the voting on KIP-126. The KIP is intended to solve
the problem that RecordTooLargeExceptions are thrown from the producer due
to inaccurate estimation of the compression ratio. The solution is to split
and resend the over sized batches if possible. A new metric is introduced
to the producer to show the batch split rate.

The KIP wiki is following:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-126
+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size

We have been running a producer with this patch for some time in our mirror
maker and it looks working fine.

Thanks,

Jiangjie (Becket) Qin

[jira] [Resolved] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeyhun Karimov resolved KAFKA-4144.
---
Resolution: Fixed
  Reviewer: Matthias J. Sax

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeyhun Karimov closed KAFKA-4144.
-

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5126) Implement KIP-98 transactional methods in the MockProducer

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5126:
---

 Summary: Implement KIP-98 transactional methods in the MockProducer
 Key: KAFKA-5126
 URL: https://issues.apache.org/jira/browse/KAFKA-5126
 Project: Kafka
  Issue Type: Test
Reporter: Apurva Mehta
Assignee: Apurva Mehta


The initial code for the transactional producer leaves the implementation of 
`initTransactions`, `beginTransaction`, `sendOffsetsToTransaction`, 
`commitTransaction`, and `abortTransaction` empty in the MockProducer. We need 
have some implementation there so that our mocks stay healthy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Dong Lin]

+1 (non-binding)

On Tue, Apr 25, 2017 at 10:42 AM, Becket Qin  wrote:

> Hi,
>
> I would like to start the voting on KIP-126. The KIP is intended to solve
> the problem that RecordTooLargeExceptions are thrown from the producer due
> to inaccurate estimation of the compression ratio. The solution is to split
> and resend the over sized batches if possible. A new metric is introduced
> to the producer to show the batch split rate.
>
> The KIP wiki is following:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-126
> +-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size
>
> We have been running a producer with this patch for some time in our mirror
> maker and it looks working fine.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

Re: [VOTE] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

[Becket Qin]

I just updated the KIP name to reflect the actual proposal. I'll close this
vote and start another one to avoid confusion.

On Tue, Apr 25, 2017 at 12:27 PM, Dong Lin  wrote:

> +1 (non-binding)
>
> On Tue, Apr 25, 2017 at 10:42 AM, Becket Qin  wrote:
>
> > Hi,
> >
> > I would like to start the voting on KIP-126. The KIP is intended to solve
> > the problem that RecordTooLargeExceptions are thrown from the producer
> due
> > to inaccurate estimation of the compression ratio. The solution is to
> split
> > and resend the over sized batches if possible. A new metric is introduced
> > to the producer to show the batch split rate.
> >
> > The KIP wiki is following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-126
> > +-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size
> >
> > We have been running a producer with this patch for some time in our
> mirror
> > maker and it looks working fine.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

[VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

[Becket Qin]

Hi,

I would like to start the voting on KIP-126. The KIP is intended to solve
the problem that RecordTooLargeExceptions are thrown from the producer due
to inaccurate estimation of the compression ratio. The solution is to split
and resend the over sized batches if possible. A new metric is introduced
to the producer to show the batch split rate.

The KIP wiki is following:
*https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68715855
*

We have been running a producer with this patch for some time in our mirror
maker and it looks working fine.

Thanks,

Jiangjie (Becket) Qin

[jira] [Commented] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983498#comment-15983498
 ] 

Matthias J. Sax commented on KAFKA-5124:


I guess is related to https://issues.apache.org/jira/browse/KAFKA-5005 -- 
assuming both issues do have the same root cause. WDYT? 

> shouldInnerLeftJoin unit test fails
> ---
>
> Key: KAFKA-5124
> URL: https://issues.apache.org/jira/browse/KAFKA-5124
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Armin Braun
> Fix For: 0.11.0.0
>
>
> Unit test on trunk gives occasional failure:
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
> shouldInnerLeftJoin FAILED
> java.lang.AssertionError: Condition not met within timeout 3. 
> Expecting 1 records from topic output- while only received 0: []
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:265)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:176)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:222)
> at 
> org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:143)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983504#comment-15983504
 ] 

Matthias J. Sax commented on KAFKA-4144:


[~jeyhunkarimov] Why did you close the PR and set the JIRA to "fixed" -- this 
did not get merged yet, or did I miss something?

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983515#comment-15983515
 ] 

Jeyhun Karimov commented on KAFKA-4144:
---

[~mjsax] I am sorry, I didn't read your email carefully. My bad.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeyhun Karimov updated KAFKA-4144:
--
Status: Reopened  (was: Closed)

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4994) Fix findbugs warning about OffsetStorageWriter#currentFlushId

[Mariam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mariam John updated KAFKA-4994:
---
Status: Open  (was: Patch Available)

> Fix findbugs warning about OffsetStorageWriter#currentFlushId
> -
>
> Key: KAFKA-4994
> URL: https://issues.apache.org/jira/browse/KAFKA-4994
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Mariam John
>  Labels: newbie
>
> We should fix the findbugs warning about 
> {{OffsetStorageWriter#currentFlushId}}
> {code}
> Multithreaded correctness Warnings
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.currentFlushId; locked 
> 83% of time
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.toFlush; locked 75% of 
> time
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2906: Kafka-4994 Fix findbug warnings about OffsetStorag...

[johnma14]

Github user johnma14 closed the pull request at:

https://github.com/apache/kafka/pull/2906


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2913: KAFKA-4994: Fix findbug warnings about OffsetStora...

[johnma14]

Github user johnma14 closed the pull request at:

https://github.com/apache/kafka/pull/2913


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5124) shouldInnerLeftJoin unit test fails

[Armin Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983539#comment-15983539
 ] 

Armin Braun commented on KAFKA-5124:


@msjax certainly looks like it at least. I only found about 30 mins for this 
today (haven't quite reached the root yet) but should be able to figure it out 
tomorrow morning after some sleep :)

What I have learnt so far is:
* I have some indication this could be a "leakish" kind of thing ... once I ran 
like 500 iterations to get the test to fail in this way, it fails like every 
10th run, while the initial 500 runs are green. I could never make it fail 
without a bunch of "warmup" runs so far.
* Not a timeout issue ... test normally takes ~ 1s -> 30s are plenty -> setting 
Long.MAX as timeout makes it go on forever.
* This would be the thread dump with Long.MAX for the timeout (in case you can 
spot something my tired eyes can't at this point :D):

{code}
"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
  at sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:-1)
  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  - locked <0x2553> (a sun.nio.ch.EPollSelectorImpl)
  - locked <0x2554> (a java.util.Collections$UnmodifiableSet)
  - locked <0x2555> (a sun.nio.ch.Util$3)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  at org.apache.kafka.common.network.Selector.select(Selector.java:493)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:302)
  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:359)
  at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:230)
  - locked <0x2556> (a 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient)
  at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:206)
  at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1055)
  at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1002)
  at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readKeyValues(IntegrationTestUtils.java:94)
  at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils$1.conditionMet(IntegrationTestUtils.java:199)
  at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:256)
  at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:207)
  at 
org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.verifyKTableKTableJoin(KTableKTableJoinIntegrationTest.java:220)
  at 
org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest.shouldInnerLeftJoin(KTableKTableJoinIntegrationTest.java:141)
  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source:-1)
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
  at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
  at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
  at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
  at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
  at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
  at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
  at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
  at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
  at org.junit.runner.JUnitCore.run(JUnitCo

[jira] [Commented] (KAFKA-4994) Fix findbugs warning about OffsetStorageWriter#currentFlushId

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983540#comment-15983540
 ] 

ASF GitHub Bot commented on KAFKA-4994:
---

Github user johnma14 closed the pull request at:

https://github.com/apache/kafka/pull/2913


> Fix findbugs warning about OffsetStorageWriter#currentFlushId
> -
>
> Key: KAFKA-4994
> URL: https://issues.apache.org/jira/browse/KAFKA-4994
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Mariam John
>  Labels: newbie
>
> We should fix the findbugs warning about 
> {{OffsetStorageWriter#currentFlushId}}
> {code}
> Multithreaded correctness Warnings
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.currentFlushId; locked 
> 83% of time
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.toFlush; locked 75% of 
> time
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983547#comment-15983547
 ] 

Matthias J. Sax commented on KAFKA-4144:


No problem :)

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
> Fix For: 0.11.0.0
>
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4144:
---
Fix Version/s: 0.11.0.0

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
> Fix For: 0.11.0.0
>
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

[Dong Lin]

+1 (non-binding)

On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin  wrote:

> Hi,
>
> I would like to start the voting on KIP-126. The KIP is intended to solve
> the problem that RecordTooLargeExceptions are thrown from the producer due
> to inaccurate estimation of the compression ratio. The solution is to split
> and resend the over sized batches if possible. A new metric is introduced
> to the producer to show the batch split rate.
>
> The KIP wiki is following:
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68715855
>  >*
>
> We have been running a producer with this patch for some time in our mirror
> maker and it looks working fine.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

[GitHub] kafka pull request #2914: KAFKA-4994: Fix findbug warnings about OffsetStora...

[johnma14]

GitHub user johnma14 opened a pull request:

https://github.com/apache/kafka/pull/2914

KAFKA-4994: Fix findbug warnings about OffsetStorageWriter

OffsetStorageWriter is not a thread-safe class and should be accessed
only from a Task's processing thread. The WorkerSourceTask class calls
the different methods (offset, beginFlush, cancelFlush, handleFinishWrite)
within a synchronized block. Hence the method definitions in
OffsetStorageWriter.java does not need to contain the keyword synchronized
again.

In the OffsetStorageWriter.java class, the doFlush() method is not 
explicitely
synchronized like the other methods in this class. Hence this can lead to
inconsistent synchronization of variables like currentFlushId and toFlush 
which
are set in the synchronized methods within this class.
- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageWriter.java
- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L295


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnma14/kafka bug/kafka-4994

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2914


commit d481c10818b59e67eaac0aa598e845773b07e51d
Author: Mariam John 
Date:   2017-04-25T20:47:04Z

KAFKA-4994: Fix findbug warnings about OffsetStorageWriter

OffsetStorageWriter is not a thread-safe class and should be accessed
only from a Task's processing thread. The WorkerSourceTask class calls
the different methods (offset, beginFlush, cancelFlush, handleFinishWrite)
within a synchronized block. Hence the method definitions in
OffsetStorageWriter.java does not need to contain the keyword synchronized
again.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4994) Fix findbugs warning about OffsetStorageWriter#currentFlushId

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983594#comment-15983594
 ] 

ASF GitHub Bot commented on KAFKA-4994:
---

GitHub user johnma14 opened a pull request:

https://github.com/apache/kafka/pull/2914

KAFKA-4994: Fix findbug warnings about OffsetStorageWriter

OffsetStorageWriter is not a thread-safe class and should be accessed
only from a Task's processing thread. The WorkerSourceTask class calls
the different methods (offset, beginFlush, cancelFlush, handleFinishWrite)
within a synchronized block. Hence the method definitions in
OffsetStorageWriter.java does not need to contain the keyword synchronized
again.

In the OffsetStorageWriter.java class, the doFlush() method is not 
explicitely
synchronized like the other methods in this class. Hence this can lead to
inconsistent synchronization of variables like currentFlushId and toFlush 
which
are set in the synchronized methods within this class.
- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageWriter.java
- 
https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L295


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/johnma14/kafka bug/kafka-4994

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2914


commit d481c10818b59e67eaac0aa598e845773b07e51d
Author: Mariam John 
Date:   2017-04-25T20:47:04Z

KAFKA-4994: Fix findbug warnings about OffsetStorageWriter

OffsetStorageWriter is not a thread-safe class and should be accessed
only from a Task's processing thread. The WorkerSourceTask class calls
the different methods (offset, beginFlush, cancelFlush, handleFinishWrite)
within a synchronized block. Hence the method definitions in
OffsetStorageWriter.java does not need to contain the keyword synchronized
again.




> Fix findbugs warning about OffsetStorageWriter#currentFlushId
> -
>
> Key: KAFKA-4994
> URL: https://issues.apache.org/jira/browse/KAFKA-4994
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Mariam John
>  Labels: newbie
>
> We should fix the findbugs warning about 
> {{OffsetStorageWriter#currentFlushId}}
> {code}
> Multithreaded correctness Warnings
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.currentFlushId; locked 
> 83% of time
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.toFlush; locked 75% of 
> time
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Jeyhun Karimov]

Dear all,

I am closing this vote now. The KIP got accepted with

+3 binding (Guozhang, Ewen, Gwen)

Thanks all (especially for Mathias) for guiding me throughout my first KIP.


Cheers,
Jeyhun

On Mon, Apr 24, 2017 at 9:32 PM Thomas Becker  wrote:

> +1 (non-binding)
>
> On Tue, 2017-02-28 at 08:59 +, Jeyhun Karimov wrote:
> > Dear community,
> >
> > I'd like to start the vote for KIP-123:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6871
> > 4788
> >
> >
> > Cheers,
> > Jeyhun
> --
>
>
> Tommy Becker
>
> Senior Software Engineer
>
> O +1 919.460.4747 <(919)%20460-4747>
>
> tivo.com
>
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
-- 
-Cheers

Jeyhun

[jira] [Commented] (KAFKA-4994) Fix findbugs warning about OffsetStorageWriter#currentFlushId

[Mariam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983633#comment-15983633
 ] 

Mariam John commented on KAFKA-4994:


[~cmccabe] [~ewencp] A fix for this issue is ready for review whenever you have 
some time. Thank you. 

> Fix findbugs warning about OffsetStorageWriter#currentFlushId
> -
>
> Key: KAFKA-4994
> URL: https://issues.apache.org/jira/browse/KAFKA-4994
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Mariam John
>  Labels: newbie
>
> We should fix the findbugs warning about 
> {{OffsetStorageWriter#currentFlushId}}
> {code}
> Multithreaded correctness Warnings
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.currentFlushId; locked 
> 83% of time
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.toFlush; locked 75% of 
> time
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4994) Fix findbugs warning about OffsetStorageWriter#currentFlushId

[Mariam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mariam John updated KAFKA-4994:
---
Status: Patch Available  (was: Open)

> Fix findbugs warning about OffsetStorageWriter#currentFlushId
> -
>
> Key: KAFKA-4994
> URL: https://issues.apache.org/jira/browse/KAFKA-4994
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Mariam John
>  Labels: newbie
>
> We should fix the findbugs warning about 
> {{OffsetStorageWriter#currentFlushId}}
> {code}
> Multithreaded correctness Warnings
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.currentFlushId; locked 
> 83% of time
> IS2_INCONSISTENT_SYNC:   Inconsistent synchronization of 
> org.apache.kafka.connect.storage.OffsetStorageWriter.toFlush; locked 75% of 
> time
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4144:
---
Status: Patch Available  (was: Reopened)

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
> Fix For: 0.11.0.0
>
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

[Bill Bejeck]

+1

On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin  wrote:

> +1 (non-binding)
>
> On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin  wrote:
>
> > Hi,
> >
> > I would like to start the voting on KIP-126. The KIP is intended to solve
> > the problem that RecordTooLargeExceptions are thrown from the producer
> due
> > to inaccurate estimation of the compression ratio. The solution is to
> split
> > and resend the over sized batches if possible. A new metric is introduced
> > to the producer to show the batch split rate.
> >
> > The KIP wiki is following:
> > *https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68715855
> >  action?pageId=68715855
> > >*
> >
> > We have been running a producer with this patch for some time in our
> mirror
> > maker and it looks working fine.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

[GitHub] kafka pull request #2911: MINOR: Improve information in assert failure for t...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/2911


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2911: MINOR: Improve information in assert failure for t...

[ijuma]

GitHub user ijuma reopened a pull request:

https://github.com/apache/kafka/pull/2911

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown

This test is failing consistently in 
https://jenkins.confluent.io/job/kafka-trunk/,
but nowhere else. I ran this branch in a clone of that job several times 
and this
test didn't fail. I suggest we merge this PR, which improves the test, to 
help us
gather more information about the test failure.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
socket-server-test-metric-collection-after-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2911


commit aba3ed482e41db011974a44de72e57bc2e3b1a7d
Author: Ismael Juma 
Date:   2017-04-24T22:12:18Z

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2911: MINOR: Improve information in assert failure for t...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/2911


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5119) Transient test failure SocketServerTest.testMetricCollectionAfterShutdown

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983801#comment-15983801
 ] 

ASF GitHub Bot commented on KAFKA-5119:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2915

KAFKA-5119: Improve information in assert failure for 
testMetricCollectionAfterShutdown

This test is failing consistently in 
https://jenkins.confluent.io/job/kafka-trunk/,
but nowhere else. I ran this branch in a clone of that job several times 
and this
test didn't fail. I suggest we merge this PR, which improves the test, to 
help us
gather more information about the test failure.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
socket-server-test-metric-collection-after-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2915


commit aba3ed482e41db011974a44de72e57bc2e3b1a7d
Author: Ismael Juma 
Date:   2017-04-24T22:12:18Z

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown




> Transient test failure SocketServerTest.testMetricCollectionAfterShutdown
> -
>
> Key: KAFKA-5119
> URL: https://issues.apache.org/jira/browse/KAFKA-5119
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Jason Gustafson
>
> From a recent build:
> {code}
> 20:04:15 kafka.network.SocketServerTest > testMetricCollectionAfterShutdown 
> FAILED
> 20:04:15 java.lang.AssertionError: expected:<0.0> but 
> was:<1.603886948862125>
> 20:04:15 at org.junit.Assert.fail(Assert.java:88)
> 20:04:15 at org.junit.Assert.failNotEquals(Assert.java:834)
> 20:04:15 at org.junit.Assert.assertEquals(Assert.java:553)
> 20:04:15 at org.junit.Assert.assertEquals(Assert.java:683)
> 20:04:15 at 
> kafka.network.SocketServerTest.testMetricCollectionAfterShutdown(SocketServerTest.scala:414)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2915: KAFKA-5119: Improve information in assert failure ...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2915

KAFKA-5119: Improve information in assert failure for 
testMetricCollectionAfterShutdown

This test is failing consistently in 
https://jenkins.confluent.io/job/kafka-trunk/,
but nowhere else. I ran this branch in a clone of that job several times 
and this
test didn't fail. I suggest we merge this PR, which improves the test, to 
help us
gather more information about the test failure.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
socket-server-test-metric-collection-after-shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2915


commit aba3ed482e41db011974a44de72e57bc2e3b1a7d
Author: Ismael Juma 
Date:   2017-04-24T22:12:18Z

MINOR: Improve information in assert failure for 
testMetricCollectionAfterShutdown




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4905) StreamPartitionAssignor doesn't respect subscriptions to assign partitions.

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983803#comment-15983803
 ] 

Matthias J. Sax commented on KAFKA-4905:


[~stevenschlansker] I just stubbled over an older comment on KAFKA-4722 and 
this seems to be related you the stack trace you did post there.

> StreamPartitionAssignor doesn't respect subscriptions to assign partitions.
> ---
>
> Key: KAFKA-4905
> URL: https://issues.apache.org/jira/browse/KAFKA-4905
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Florian Hussonnois
>
> Both RangeAssignor and RoundRobinAssignor use the subscriptions to assign 
> partition to each consumer. This allow to have two consumers belonging to the 
> the same group and subscribing to two differents topics.
> This doesn't seem to be the case of the StreamPartitionAssignor resulting to 
> an IllegalArgumentException thrown during rebalance. 
> java.lang.IllegalArgumentException: Assigned partition foo-2 for 
> non-subscribed topic regex pattern; subscription pattern is bar
>   at 
> org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:190)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:216)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:352)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> This is because the consumer group leader attempt to assign partitions to a 
> consumer that didn't subscribe to the associated topic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

[James Cheng]

Having "MarkedForDeletion" before "Configs" may break anyone who is parsing 
this output, since they may be expecting the 4th string to be "Configs".

I know that the Compatibility section already says that people parsing this may 
have to adjust their parsing logic, so maybe that covers my concern already. 
But inserting the new MarkedForDeletion word into the middle of the string 
seems like it'll break parsing more than just adding a new value at the end.

I'm fine either way, though.

-James

> On Apr 25, 2017, at 9:38 AM, Vahid S Hashemian  
> wrote:
> 
> Thanks for the KIP Mickael. 
> Looks good. I also prefer 'MarkedForDeletion' before 'Configs'.
> 
> --Vahid
> 
> 
> 
> From:   Ismael Juma 
> To: dev@kafka.apache.org
> Date:   04/25/2017 04:15 AM
> Subject:Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to 
> show topics marked for deletion
> Sent by:isma...@gmail.com
> 
> 
> 
> Thanks for the KIP. Would it make sense for MarkedForDeletion to be before
> `Configs`? I can see arguments both ways, so I was wondering what your
> thoughts were?
> 
> Ismael
> 
> On Thu, Mar 30, 2017 at 5:39 PM, Mickael Maison 
> wrote:
> 
>> Hi all,
>> 
>> We created KIP-137: Enhance TopicCommand --describe to show topics
>> marked for deletion
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 
> 137%3A+Enhance+TopicCommand+--describe+to+show+topics+marked+for+deletion
>> 
>> Please help review the KIP. You feedback is appreciated!
>> 
>> Thanks
>> 
> 
> 
> 
> 

[GitHub] kafka pull request #2916: WIP: Avoid FileInputStream and FileOutputStream

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2916

WIP: Avoid FileInputStream and FileOutputStream



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka no-file-input-stream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2916


commit ff8a9cffc9e3cca2185148a58b6f4d06fbe69e94
Author: Ismael Juma 
Date:   2017-03-30T14:52:26Z

MINOR: Avoid FileInputStream and FileOutputStream

They rely on finalizers, which create unnecessary GC
load. The alternatives are as easy to use and don't
have this issue.

Also use FileChannel directly instead of retrieving
it from RandomAccessFile.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2895: KAFKA-5111: Improve internal Task APIs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5111) Improve internal Task APIs

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-5111:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2895
[https://github.com/apache/kafka/pull/2895]

> Improve internal Task APIs
> --
>
> Key: KAFKA-5111
> URL: https://issues.apache.org/jira/browse/KAFKA-5111
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.11.0.0
>
>
> Currently, the internal interface for tasks is not very clean and it's hard 
> to reason about the control flow when tasks get closes, suspended, resumed 
> etc. This makes exception handling particularly hard.
> We want to refactor this part of the code to get a clean control flow and 
> interface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5111) Improve internal Task APIs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983955#comment-15983955
 ] 

ASF GitHub Bot commented on KAFKA-5111:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2895


> Improve internal Task APIs
> --
>
> Key: KAFKA-5111
> URL: https://issues.apache.org/jira/browse/KAFKA-5111
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.11.0.0
>
>
> Currently, the internal interface for tasks is not very clean and it's hard 
> to reason about the control flow when tasks get closes, suspended, resumed 
> etc. This makes exception handling particularly hard.
> We want to refactor this part of the code to get a clean control flow and 
> interface.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1455

[Apache Jenkins Server]

See 


Changes:

[becket.qin] KAFKA-4840; BufferPool errors can cause buffer pool to go into a 
bad

[wangguoz] KAFKA-5111: Improve internal Task APIs of Streams

--
[...truncated 1.47 MB...]

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroupWithOffsetsOnly 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroupWithOffsetsOnly 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.ProducerBounceTest > testBrokerFailure STARTED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms STARTED

kafka.api.SaslMultiMechanismConsumerTest > testMultipleBrokerMechanisms PASSED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslMultiMechanismConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslMultiMechanismConsumerTest > testSimpleConsumption PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero STARTED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextP

[GitHub] kafka pull request #2917: Kafka 5111 code cleanup follow up

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2917

Kafka 5111 code cleanup follow up

 - mainly moving methods
 - also improved logging

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-5111-code-cleanup-follow-up

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2917


commit 84daf7ac8950cef736bc9458b5ec0e844827264a
Author: Matthias J. Sax 
Date:   2017-04-22T04:57:08Z

KAFA-5111: Improve internal Task API (code cleanup follow up)

commit 6e4068df9c81ade6ed5e812a219a8087e194dfb8
Author: Matthias J. Sax 
Date:   2017-04-25T18:59:24Z

Improve logging




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-123: Allow per stream/table timestamp extractor

[Michael Noll]

Thanks for your work and for driving this, Jeyhun! :-)

-Michael


On Tue, Apr 25, 2017 at 11:11 PM, Jeyhun Karimov 
wrote:

> Dear all,
>
> I am closing this vote now. The KIP got accepted with
>
> +3 binding (Guozhang, Ewen, Gwen)
>
> Thanks all (especially for Mathias) for guiding me throughout my first KIP.
>
>
> Cheers,
> Jeyhun
>
> On Mon, Apr 24, 2017 at 9:32 PM Thomas Becker  wrote:
>
> > +1 (non-binding)
> >
> > On Tue, 2017-02-28 at 08:59 +, Jeyhun Karimov wrote:
> > > Dear community,
> > >
> > > I'd like to start the vote for KIP-123:
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=6871
> > > 4788
> > >
> > >
> > > Cheers,
> > > Jeyhun
> > --
> >
> >
> > Tommy Becker
> >
> > Senior Software Engineer
> >
> > O +1 919.460.4747 <(919)%20460-4747>
> >
> > tivo.com
> >
> >
> > 
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
> --
> -Cheers
>
> Jeyhun
>

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Michael Noll (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984230#comment-15984230
 ] 

Michael Noll commented on KAFKA-4144:
-

As part of this work we should also update the respective Kafka docs.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api, kip
> Fix For: 0.11.0.0
>
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} 
> value to {{KafkaStreams}}. That means you can only have a single timestamp 
> extractor per app, even though you may be joining multiple streams/tables 
> that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> {{KStreamBuilder.stream()/table()}}, just like you can specify key and value 
> serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but 
> not for intermediate topics. For PAPI we cannot enforce this, but for DSL 
> {{through()}} should not allow to set a custom extractor by the user. In 
> contrast, with regard to KAFKA-4785, is must internally set an extractor that 
> returns the record's metadata timestamp in order to overwrite the global 
> extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}). This change should be done in 
> KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)