Re: Using KIP Process in Apache Flink

[Aljoscha Krettek]

Thanks, appreciate it! :-)

On Thu, 7 Jul 2016 at 17:21 Ismael Juma  wrote:

> Hi Aljoscha,
>
> Thanks for sharing your intention to use a process similar to our KIP
> process. You are more than welcome to copy the structures and docs that we
> have for the KIP process. :)
>
> Ismael
>
> On Thu, Jul 7, 2016 at 4:16 PM, Aljoscha Krettek 
> wrote:
>
> > Hi Kafka Community,
> > I'm a member of the Flink community and I would like to ask if you're
> okay
> > with us using your KIP process for Flink. We are at a stage where we have
> > to be more careful in planning big feature additions to Flink and are
> > looking for ways to formalize our processes there. I feel we don't have
> to
> > reinvent the wheel if other projects already put thought into this and
> > established a process that works.
> >
> > So, would you be okay with us copying the structures and docs that you
> have
> > in place for the KIP process? With adaptions to suit Flink, of course.
> >
> > Best,
> > Aljoscha
> >
>

Re: SSL: Broker works but consumer/producer fail

[Narendra Bidari]

Hi Vineet,

The setup of ssl Kafka requires to make one too many steps precise correct. I 
have listed some below . Hope it helps 
https://github.com/Symantec/kafka-security-0.9

Sent from my iPhone

Regards

> On Jul 7, 2016, at 4:49 PM, Vineet Kumar  wrote:
> 
> Hi
>  I followed Apache Kafka SSL instructions verbatim but my producer and
> consumer both hang or error out as follows.
> openssl s_client BTW does work fine with the server below yielding
> certificates etc thereby confirming that the server can talk back SSL.
> 
> 
> *Producer and Consumer*
> =
> 
> Config changes (client-ssl.properties)
> ---
> security.protocol=SSL
> 
> % bin/kafka-console-*consumer*.sh --bootstrap-server 192.168.1.XXX:9093
> --topic test --new-consumer --consumer.config config/client-ssl.properties
> **
> 
> % bin/kafka-console-*producer*.sh --broker-list 192.168.1.XXX:9093 --topic
> test --producer.config config/client-ssl.properties
> a
> 
> **
> 
> [2016-07-07 16:35:57,670] ERROR Error when sending message to topic test
> with key: null, value: 29 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata
> after 6 ms.
> 
> *Broker*
> ==
> Config changes (server.properties)
> ---
> listeners=SSL://192.168.1.XXX:9093
> security.inter.broker.protocol=SSL
> advertised.listeners=SSL://192.168.1.XXX:9093
> ssl.keystore.location=/<..>/server.keystore.jks
> ssl.keystore.password=
> ssl.key.password=
> 
> % bin/kafka-*server*-start.sh config/server.properties
> [2016-07-07 16:14:00,805] INFO Registered broker 0 at path /brokers/ids/0
> with addresses: *SSL -> EndPoint(192.168.1.XXX,9093,SSL)*
> (kafka.utils.ZkUtils)
> [2016-07-07 16:14:00,820] INFO New leader is 0
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2016-07-07 16:14:00,825] INFO Kafka version : 0.10.0.0
> (org.apache.kafka.common.utils.AppInfoParser)
> [2016-07-07 16:14:00,825] INFO Kafka commitId : b8642491e78c5a13
> (org.apache.kafka.common.utils.AppInfoParser)
> [2016-07-07 16:14:00,827] INFO [Kafka Server 0], started
> (kafka.server.KafkaServer)
> 
> *Zookeeper*
> =
> Config changes
> ---
>  Nothing
> 
> % bin/zookeeper-server-start.sh config/zookeeper.properties
> 
> 
> [2016-07-07 16:13:18,002] INFO binding to port 0.0.0.0/0.0.0.0:2181
> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> 
> 
> [2016-07-07 16:14:00,131] INFO Accepted socket connection from /
> 127.0.0.1:41188 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> [2016-07-07 16:14:00,189] INFO Client attempting to establish new session
> at /127.0.0.1:41188 (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-07-07 16:14:00,199] INFO Established session 0x155c7a306dc with
> negotiated timeout 6000 for client /127.0.0.1:41188
> (org.apache.zookeeper.server.ZooKeeperServer)
> [2016-07-07 16:14:00,652] INFO Got user-level KeeperException when
> processing sessionid:0x155c7a306dc type:delete cxid:0x22 zxid:0xd6
> txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election
> Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election
> (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-07-07 16:14:00,778] INFO Got user-level KeeperException when
> processing sessionid:0x155c7a306dc type:create cxid:0x29 zxid:0xd7
> txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode =
> NodeExists for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
> [2016-07-07 16:14:00,778] INFO Got user-level KeeperException when
> processing sessionid:0x155c7a306dc type:create cxid:0x2a zxid:0xd8
> txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode =
> NodeExists for /brokers/ids
> (org.apache.zookeeper.server.PrepRequestProcessor)

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Eno Thereska]

+1 (non-binding)

> On 7 Jul 2016, at 18:31, Sriram Subramanian  wrote:
> 
> +1
> 
> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai 
> wrote:
> 
>> +1
>> 
>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy 
>> wrote:
>>> 
 Thanks Henry - we've updated the KIP with an example and the new config
 parameter required. FWIW the user doesn't register a listener, they
>>> provide
 a host:port in config. It is expected they will start a service running
>>> on
 that host:port that they can use to connect to the running KafkaStreams
 Instance.
 
 Thanks,
 Damian
 
 On Thu, 7 Jul 2016 at 06:06 Henry Cai 
>>> wrote:
 
> It wasn't quite clear to me how the user program interacts with the
> discovery API, especially on the user supplied listener part, how
>> does
 the
> user program supply that listener to KafkaStreams and how does
 KafkaStreams
> know which port the user listener is running, maybe a more complete
> end-to-end example including the steps on registering the user
>> listener
 and
> whether the user listener needs to be involved with task
>> reassignment.
> 
> 
> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang 
 wrote:
> 
>> ＋1
>> 
>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy 
> wrote:
>> 
>>> Hi all,
>>> 
>>> I'd like to initiate the voting process for KIP-67
>>> <
>>> 
>> 
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
 
>>> 
>>> KAFKA-3909  is
>>> the
> top
>>> level JIRA for this effort.
>>> 
>>> Initial PRs for Step 1 of the process are:
>>> Expose State Store Names <
>>> https://github.com/apache/kafka/pull/1526>
> and
>>> Query Local State Stores <
>>> https://github.com/apache/kafka/pull/1565>
>>> 
>>> Thanks,
>>> Damian
>>> 
>> 
>> 
>> 
>> --
>> -- Guozhang
>> 
> 
 
>>> 
>>> 
>>> 
>>> --
>>> Best regards,
>>> Michael Noll
>>> 
>>> 
>>> 
>>> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
>>> Apache Kafka and Confluent Platform: www.confluent.io/download
>>> *
>>> 
>> 

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[Andy Coates (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367489#comment-15367489
 ] 

Andy Coates commented on KAFKA-3919:


[~junrao] Yes, we lost a good number of brokers in a power outage.

The solution you're proposing in KAFKA-1211 looks fairly involved, i.e. a 
protocol change, is this something you think I can pick up, (having never 
committed to Kafka, but no newbie to distributed programming), or something I'd 
be best of leaving to the committers? (I'm conscious that its been sat as a 
known issue for a long time, and internally this is viewed as a blocker...)



> Broker faills to start after ungraceful shutdown due to non-monotonically 
> incrementing offsets in logs
> --
>
> Key: KAFKA-3919
> URL: https://issues.apache.org/jira/browse/KAFKA-3919
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1
>Reporter: Andy Coates
>
> Hi All,
> I encountered an issue with Kafka following a power outage that saw a 
> proportion of our cluster disappear. When the power came back on several 
> brokers halted on start up with the error:
> {noformat}
>   Fatal error during KafkaServerStartable startup. Prepare to shutdown”
>   kafka.common.InvalidOffsetException: Attempt to append an offset 
> (1239742691) to position 35728 no larger than the last offset appended 
> (1239742822) to 
> /data3/kafka/mt_xp_its_music_main_itsevent-20/001239444214.index.
>   at 
> kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:207)
>   at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
>   at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at kafka.log.OffsetIndex.append(OffsetIndex.scala:197)
>   at kafka.log.LogSegment.recover(LogSegment.scala:188)
>   at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
>   at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:160)
>   at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>   at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>   at kafka.log.Log.loadSegments(Log.scala:160)
>   at kafka.log.Log.(Log.scala:90)
>   at 
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:150)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The only way to recover the brokers was to delete the log files that 
> contained non monotonically incrementing offsets.
> We've spent some time digging through the logs and I feel I may have worked 
> out the sequence of events leading to this issue, (though this is based on 
> some assumptions I've made about the way Kafka is working, which may be 
> wrong).
> First off, we have unclean leadership elections disable. (We did later enable 
> them to help get around some other issues we were having, but this was 
> several hours after this issue manifested), and we're producing to the topic 
> with gzip compression and acks=1
> We looked through the data logs that were causing the brokers to not start. 
> What we found the initial part of the log has monotonically increasing 
> offset, where each compressed batch normally contained one or two records. 
> Then the is a batch that contains many records, whose first records have an 
> offset below the previous batch and whose last record has an offset above the 
> previous batch. Following on from this there continues a period of large 
> batches, with monotonically increasing offsets, and then the log returns to 
> batches with one or two records.
> Our working assumption here is that the period before the offset dip, with 
> the small batches, is pre-outage normal operation. The period of larger 
> batches is from just after the outage, where producers have a back log to 
> processes when the partition becomes available, and then things return to 
> normal batch sizes again once the back log clears.
> We did also look through the Kafka's application log

Re: [DISCUSS] Client Side Auto Topic Creation

[Tommy Becker]

I think the use case for not blowing up the consumer is simply to not create an 
implicit ordering in which your services have to come up.

On 07/07/2016 04:47 PM, Jun Rao wrote:

It seems that it makes sense for the writer to trigger auto topic creation,
but not the reader. So, my preference is Jay's option #1: add a new
configuration to enable topic creation on the producer side and defaults to
true. If the topic doesn't exist, the producer will send a
createTopicRequest and pick up the broker side defaults for #partitions and
replication factor. This matches the current behavior and won't surprise
people. People who want to enforce manual topic creation can disable auto
topic creation on the producer.

On the consumer side, throwing an exception to the client when a topic
doesn't exist probably works for most cases. I am wondering if there is a
case where a user really wants to start the consumer before the topic is
created.

Thanks,

Jun


On Fri, Jul 1, 2016 at 4:09 AM, Ismael Juma 
 wrote:



Hi all,

I think there are a few things being discussed and it would be good to make
that explicit:

1. If and how we expose auto-topic creation in the client (under the
assumption that the server auto-topic creation will be deprecated and
eventually removed)
2. The ability to create topics with the cluster defaults for replication
factor and partition counts
3. Support for topic "specs"
4. The fact that some exceptions are retriable in some cases, but not
others

My thoughts on each:

1. I prefer the approach where we throw an exception and let the clients
create the topic via `AdminClient` if that's what they need.
2. Like Grant, I'm unsure that will generally be used in a positive way.
However, if this is what we need to be able to deprecate server auto-topic
creation, the benefits outweigh the costs in my opinion.
3. Something like this would be good to have and could potentially provide
a better solution than 2. However, it needs a separate KIP and may take a
while for the final design to be agreed. So, it should not prevent progress
from being made in my opinion.
4. This has come up before. Encoding whether an exception is retriable or
not via inheritance is a bit restrictive. Also, something that should be
discussed separately, probably.

Ismael

On Wed, Jun 29, 2016 at 10:37 PM, Grant Henke 
 wrote:



Hi Roger and Constantine,

Thanks for the feedback.

I agree that configuration to maintain guarantees is commonly spread


across


enterprise teams, making it difficult to get right. That said its also


hard


to solve for every company structure too. I think there is room for an


open


discussion about what configs should be able to be
validated/enforced/overridden and where configurations should live. I


think


thats big enough for a whole new KIP and would like to push that


discussion


out until that KIP is opened. These discussions would also make sense in
KIP-37
- Add Namespaces to Kafka
<



https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka


.


To ensure we allow validation and overrides at the namespace level.

That said, KIP-4 will be introducing a config request/response protocol
and adding call to get/alter configs to the admin api. You could


leverage


that to do some of the client validation and defaulting based on your
needs. Look for a discussion thread from me on that soon.

As far as auto topic creation goes, it sounds like failing fast and
allowing the client application to create the topic would provide the


most


flexibility to ensure the topic matches its needed specifications.

Thanks,
Grant

On Wed, Jun 29, 2016 at 3:02 PM, Konstantin Zadorozhny <
konstantin.zadoroz...@tubemogul.com>
 wrote:



Roger,

I concur with everything you said.

Couple more use cases to prove the point:

  1. Some topics should always have 1 and only one partition.
  2. CDC application based on Kafka Connect. Those type of application
  absolutely must know how to create properly configured topics:
compacted, 1
  partition, replication factor 3, 2 min in sync replicas. In many


cases


per
  table or per database configuration overrides will be useful too.

If producer and consumer are able to verify topic configuration on


startup


would be really useful. A spec would be great way to document the


intent


of


the code. A lot of silly (but quite hard to pin down) production issues
could have been prevented by having producer to fail fast on


misconfigured


topics.

To add to the auto-creation configuration tally. We do have topic
auto-creation disabled on all our clusters.

*Konstantin Zadorozhny*
www.tubemogul.com

On Wed, Jun 29, 2016 at 11:17 AM, Roger Hoover 
mailto:roger.hoo...@gmail.com>





wrote:



My comments go a bit beyond just topic creation but I'd like to see


Kafka


make it easier for application developers to s

[GitHub] kafka pull request #1598: KAFKA-3933: close deepIterator during log recovery

[tcrayford]

GitHub user tcrayford opened a pull request:

https://github.com/apache/kafka/pull/1598

KAFKA-3933: close deepIterator during log recovery

Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.

Introduces `kafka.common.ClosableIterator`, which is an iterator that
can be closed, and changes the signature of
`ByteBufferMessageSet.deepIterator` to return it, then changes the
caller `LogSegment` to always close the iterator.

https://issues.apache.org/jira/browse/KAFKA-3933

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/heroku/kafka dont_leak_native_memory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1598.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1598


commit 06e748f4cc7dd8c2a860bd938b535e8172e1
Author: Tom Crayford 
Date:   2016-07-08T11:50:21Z

KAFKA-3933: close deepIterator during log recovery

Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.

Introduces `kafka.common.ClosableIterator`, which is an iterator that
can be closed, and changes the signature of
`ByteBufferMessageSet.deepIterator` to return it, then changes the
caller `LogSegment` to always close the iterator.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367613#comment-15367613
 ] 

ASF GitHub Bot commented on KAFKA-3933:
---

GitHub user tcrayford opened a pull request:

https://github.com/apache/kafka/pull/1598

KAFKA-3933: close deepIterator during log recovery

Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.

Introduces `kafka.common.ClosableIterator`, which is an iterator that
can be closed, and changes the signature of
`ByteBufferMessageSet.deepIterator` to return it, then changes the
caller `LogSegment` to always close the iterator.

https://issues.apache.org/jira/browse/KAFKA-3933

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/heroku/kafka dont_leak_native_memory

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1598.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1598


commit 06e748f4cc7dd8c2a860bd938b535e8172e1
Author: Tom Crayford 
Date:   2016-07-08T11:50:21Z

KAFKA-3933: close deepIterator during log recovery

Avoids leaking native memory and hence crashing brokers on bootup due to
running out of memory.

Introduces `kafka.common.ClosableIterator`, which is an iterator that
can be closed, and changes the signature of
`ByteBufferMessageSet.deepIterator` to return it, then changes the
caller `LogSegment` to always close the iterator.




> Kafka OOM During Log Recovery Due to Leaked Native Memory
> -
>
> Key: KAFKA-3933
> URL: https://issues.apache.org/jira/browse/KAFKA-3933
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Linux, latest oracle java-8
>Reporter: Tom Crayford
>Assignee: Tom Crayford
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> Hi there. We've been tracking an issue where Kafka hits an 
> java.lang.OutOfMemoryError during log recovery.
> After a bunch of tracking work, we've realized we've hit an instance of a 
> long known issue: http://www.evanjones.ca/java-native-leak-bug.html
> TLDR: Kafka breaks the rule "Always close GZIPInputStream and 
> GZIPOutputStream since they use native memory via zlib" from that article.
> As such, during broker startup, when you're recovering log segments that have 
> been compressed with gzip, Kafka leaks `GZIPInputStream` all over the place.
> Our crashes during startup have this profile - the JVM heap is empty (a few 
> hundred MB), but the offheap memory is full of allocations caused by 
> `Java_java_util_zip_Deflater_init` and `deflatInit2`.
> This leads to broker crashes during startup. The only real mitigation is 
> having *far* more memory than you need to boot (which I'd guess is why folk 
> haven't noticed this in production that much yet).
> To dig into the code more (this is based on trunk). Log recovery on unflushed 
> segments eventually calls `LogSegment.recover`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L172
> On compressed segments, that leads to a call to `deepIterator`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L189
> That leads to a call to `CompressionFactory`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L95
>  which creates a `GZIPInputStream`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/CompressionFactory.scala#L46
> That `GZIPInputStream` is never closed anywhere, and the low *heap* pressure 
> means that the finalizer on `GZIPInputStream` that deallocates the native 
> buffers is never called, because GC is never triggered. Instead, we just 
> exhaust the offheap memory and then Kafka dies from an OutOfMemory error.
> Kafka *does* trigger an `inputstream.close()` call, but only when *fully* 
> reading the whole input stream (see 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L156).
>  When it's performing log recovery, in 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L189
>  it doesn't read to the end of the stream, but instead reads the first offset 
> and leaves things alone.
> This issue likely impacts `lz4` and `snappy` compressed topics in exactly the 
> same way. I think (but haven't 100% verified) that it impacts all versions of 
> Kafka that are supported (0.8 -> 0.10).
> Fixing this seems 

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

[Tom Crayford (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367625#comment-15367625
 ] 

Tom Crayford commented on KAFKA-3933:
-

Ismael: I've pushed the PR here: https://github.com/apache/kafka/pull/1598. For 
now, I've only fixed the precise memory issue that has been causing us notable 
production issues. I'm happy picking up other parts of the codebase where this 
can happen, but would rather get feedback on the first part of the approach for 
now (this is my first time contributing code to Kafka). I couldn't see a good 
or easy way to write any unit tests for this code right now.

> Kafka OOM During Log Recovery Due to Leaked Native Memory
> -
>
> Key: KAFKA-3933
> URL: https://issues.apache.org/jira/browse/KAFKA-3933
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Linux, latest oracle java-8
>Reporter: Tom Crayford
>Assignee: Tom Crayford
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> Hi there. We've been tracking an issue where Kafka hits an 
> java.lang.OutOfMemoryError during log recovery.
> After a bunch of tracking work, we've realized we've hit an instance of a 
> long known issue: http://www.evanjones.ca/java-native-leak-bug.html
> TLDR: Kafka breaks the rule "Always close GZIPInputStream and 
> GZIPOutputStream since they use native memory via zlib" from that article.
> As such, during broker startup, when you're recovering log segments that have 
> been compressed with gzip, Kafka leaks `GZIPInputStream` all over the place.
> Our crashes during startup have this profile - the JVM heap is empty (a few 
> hundred MB), but the offheap memory is full of allocations caused by 
> `Java_java_util_zip_Deflater_init` and `deflatInit2`.
> This leads to broker crashes during startup. The only real mitigation is 
> having *far* more memory than you need to boot (which I'd guess is why folk 
> haven't noticed this in production that much yet).
> To dig into the code more (this is based on trunk). Log recovery on unflushed 
> segments eventually calls `LogSegment.recover`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L172
> On compressed segments, that leads to a call to `deepIterator`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L189
> That leads to a call to `CompressionFactory`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L95
>  which creates a `GZIPInputStream`: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/CompressionFactory.scala#L46
> That `GZIPInputStream` is never closed anywhere, and the low *heap* pressure 
> means that the finalizer on `GZIPInputStream` that deallocates the native 
> buffers is never called, because GC is never triggered. Instead, we just 
> exhaust the offheap memory and then Kafka dies from an OutOfMemory error.
> Kafka *does* trigger an `inputstream.close()` call, but only when *fully* 
> reading the whole input stream (see 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L156).
>  When it's performing log recovery, in 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogSegment.scala#L189
>  it doesn't read to the end of the stream, but instead reads the first offset 
> and leaves things alone.
> This issue likely impacts `lz4` and `snappy` compressed topics in exactly the 
> same way. I think (but haven't 100% verified) that it impacts all versions of 
> Kafka that are supported (0.8 -> 0.10).
> Fixing this seems relatively annoying, but only because of some "small 
> matters of coding", nothing hugely problematic.
> The main issue is that `deepIterator` only returns an `Iterator`, which 
> doesn't have a `close()` method of any kind. We could create a new 
> `ClosableIterator` trait and have it extend Java's `AutoCloseable` 
> (https://docs.oracle.com/javase/7/docs/api/java/lang/AutoCloseable.html), 
> then explicitly call `close()` everywhere we use a `deepIterator()` and don't 
> always read to the end. Scala unfortunately doesn't seem to have a built in 
> version of Java's `try-with-resources` statement, but we can explicitly call 
> close everywhere perfectly happily.
> Another (but much more hacky) solution would be to always read to the end of 
> the iterator in `LogSegment.recover`, but that seems pretty bad, using far 
> more resources than is needed during recovery.
> I can't think of any other reasonable solutions for now, but would love to 
> hear input from the community.
> We're happy doing the work of developing a patch, but 

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Matthias J. Sax]

+1

On 07/08/2016 11:03 AM, Eno Thereska wrote:
> +1 (non-binding)
> 
>> On 7 Jul 2016, at 18:31, Sriram Subramanian  wrote:
>>
>> +1
>>
>> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai 
>> wrote:
>>
>>> +1
>>>
>>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll  wrote:
>>>
 +1 (non-binding)

 On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy 
>>> wrote:

> Thanks Henry - we've updated the KIP with an example and the new config
> parameter required. FWIW the user doesn't register a listener, they
 provide
> a host:port in config. It is expected they will start a service running
 on
> that host:port that they can use to connect to the running KafkaStreams
> Instance.
>
> Thanks,
> Damian
>
> On Thu, 7 Jul 2016 at 06:06 Henry Cai 
 wrote:
>
>> It wasn't quite clear to me how the user program interacts with the
>> discovery API, especially on the user supplied listener part, how
>>> does
> the
>> user program supply that listener to KafkaStreams and how does
> KafkaStreams
>> know which port the user listener is running, maybe a more complete
>> end-to-end example including the steps on registering the user
>>> listener
> and
>> whether the user listener needs to be involved with task
>>> reassignment.
>>
>>
>> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang 
> wrote:
>>
>>> ＋1
>>>
>>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy 
>> wrote:
>>>
 Hi all,

 I'd like to initiate the voting process for KIP-67
 <

>>>
>>
>

>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
>

 KAFKA-3909  is
 the
>> top
 level JIRA for this effort.

 Initial PRs for Step 1 of the process are:
 Expose State Store Names <
 https://github.com/apache/kafka/pull/1526>
>> and
 Query Local State Stores <
 https://github.com/apache/kafka/pull/1565>

 Thanks,
 Damian

>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>



 --
 Best regards,
 Michael Noll



 *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
 Apache Kafka and Confluent Platform: www.confluent.io/download
 *

>>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] Client Side Auto Topic Creation

[Ismael Juma]

Hi Jun,

I agree that it's closer to the existing behaviour, which some people may
be used to by now. However, I am not sure that it won't surprise people. As
Grant said, auto-topic creation is a common source of confusion and it
interacts badly with topic deletion.

If we need to provide auto-topic creation in the client as a migration path
for people who rely on it and so that we can remove the server based one
(after a suitable deprecation period), then can we at least have it false
by default? This way it's more likely that people who enable it would be
aware of the pitfalls and it would reduce the number of confused users.

Ismael

On Thu, Jul 7, 2016 at 9:47 PM, Jun Rao  wrote:

> It seems that it makes sense for the writer to trigger auto topic creation,
> but not the reader. So, my preference is Jay's option #1: add a new
> configuration to enable topic creation on the producer side and defaults to
> true. If the topic doesn't exist, the producer will send a
> createTopicRequest and pick up the broker side defaults for #partitions and
> replication factor. This matches the current behavior and won't surprise
> people. People who want to enforce manual topic creation can disable auto
> topic creation on the producer.
>
> On the consumer side, throwing an exception to the client when a topic
> doesn't exist probably works for most cases. I am wondering if there is a
> case where a user really wants to start the consumer before the topic is
> created.
>
> Thanks,
>
> Jun
>
>
> On Fri, Jul 1, 2016 at 4:09 AM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > I think there are a few things being discussed and it would be good to
> make
> > that explicit:
> >
> > 1. If and how we expose auto-topic creation in the client (under the
> > assumption that the server auto-topic creation will be deprecated and
> > eventually removed)
> > 2. The ability to create topics with the cluster defaults for replication
> > factor and partition counts
> > 3. Support for topic "specs"
> > 4. The fact that some exceptions are retriable in some cases, but not
> > others
> >
> > My thoughts on each:
> >
> > 1. I prefer the approach where we throw an exception and let the clients
> > create the topic via `AdminClient` if that's what they need.
> > 2. Like Grant, I'm unsure that will generally be used in a positive way.
> > However, if this is what we need to be able to deprecate server
> auto-topic
> > creation, the benefits outweigh the costs in my opinion.
> > 3. Something like this would be good to have and could potentially
> provide
> > a better solution than 2. However, it needs a separate KIP and may take a
> > while for the final design to be agreed. So, it should not prevent
> progress
> > from being made in my opinion.
> > 4. This has come up before. Encoding whether an exception is retriable or
> > not via inheritance is a bit restrictive. Also, something that should be
> > discussed separately, probably.
> >
> > Ismael
> >
> > On Wed, Jun 29, 2016 at 10:37 PM, Grant Henke 
> wrote:
> >
> > > Hi Roger and Constantine,
> > >
> > > Thanks for the feedback.
> > >
> > > I agree that configuration to maintain guarantees is commonly spread
> > across
> > > enterprise teams, making it difficult to get right. That said its also
> > hard
> > > to solve for every company structure too. I think there is room for an
> > open
> > > discussion about what configs should be able to be
> > > validated/enforced/overridden and where configurations should live. I
> > think
> > > thats big enough for a whole new KIP and would like to push that
> > discussion
> > > out until that KIP is opened. These discussions would also make sense
> in
> > > KIP-37
> > > - Add Namespaces to Kafka
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> > > >.
> > > To ensure we allow validation and overrides at the namespace level.
> > >
> > > That said, KIP-4 will be introducing a config request/response protocol
> > >  and adding call to get/alter configs to the admin api. You could
> > leverage
> > > that to do some of the client validation and defaulting based on your
> > > needs. Look for a discussion thread from me on that soon.
> > >
> > > As far as auto topic creation goes, it sounds like failing fast and
> > > allowing the client application to create the topic would provide the
> > most
> > > flexibility to ensure the topic matches its needed specifications.
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Wed, Jun 29, 2016 at 3:02 PM, Konstantin Zadorozhny <
> > > konstantin.zadoroz...@tubemogul.com> wrote:
> > >
> > > > Roger,
> > > >
> > > > I concur with everything you said.
> > > >
> > > > Couple more use cases to prove the point:
> > > >
> > > >1. Some topics should always have 1 and only one partition.
> > > >2. CDC application based on Kafka Connect. Those type of
> application
> > > >absolutely must know how to create properly configured topics:
> > > > compacted, 1
> > > >parti

[jira] [Created] (KAFKA-3936) Validate user parameters as early as possible

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-3936:
--

 Summary: Validate user parameters as early as possible
 Key: KAFKA-3936
 URL: https://issues.apache.org/jira/browse/KAFKA-3936
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax
Assignee: Guozhang Wang
Priority: Minor


Currently, parameters handed in by the user via public API, are not validated. 
For example {{stream.to(null)}} would fail when the underlying producer gets 
instantiated. This result in a stack trace from deep down in the library, 
making it hard to reason about the problem for the user.

We want to check all given user parameters as early as possible and raise 
corresponding (and helpful!) exceptions to explain users what the problem is, 
and how to fix it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3936) Validate user parameters as early as possible

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-3936:
---
Description: 
Currently, parameters handed in by the user via public API, are not validated. 
For example {{stream.to(null)}} would fail when the underlying producer gets 
instantiated. This result in a stack trace from deep down in the library, 
making it hard to reason about the problem for the user.

We want to check all given user parameters as early as possible and raise 
corresponding (and helpful!) exceptions to explain users what the problem is, 
and how to fix it. All parameter checks should get unit tested.

  was:
Currently, parameters handed in by the user via public API, are not validated. 
For example {{stream.to(null)}} would fail when the underlying producer gets 
instantiated. This result in a stack trace from deep down in the library, 
making it hard to reason about the problem for the user.

We want to check all given user parameters as early as possible and raise 
corresponding (and helpful!) exceptions to explain users what the problem is, 
and how to fix it.


> Validate user parameters as early as possible
> -
>
> Key: KAFKA-3936
> URL: https://issues.apache.org/jira/browse/KAFKA-3936
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>Priority: Minor
>  Labels: beginner, newbie, starter
>
> Currently, parameters handed in by the user via public API, are not 
> validated. For example {{stream.to(null)}} would fail when the underlying 
> producer gets instantiated. This result in a stack trace from deep down in 
> the library, making it hard to reason about the problem for the user.
> We want to check all given user parameters as early as possible and raise 
> corresponding (and helpful!) exceptions to explain users what the problem is, 
> and how to fix it. All parameter checks should get unit tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Michael Noll]

I have one further comment about `StreamsConfig.USER_ENDPOINT_CONFIG`.

I think we should consider to not restricting the value of this setting to
only `host:port` pairs.  By design, this setting is capturing user-driven
metadata to define an endpoint, so why restrict the creativity or
flexibility of our users?  I can imagine, for example, that users would
like to set values such as `https://host:port/api/rest/v1/` in this field
(e.g. being able to distinguish between `.../v1/` and `.../v2/` may help in
scenarios such as rolling upgrades, where, during the upgrade, older
instances may need to coexist with newer instances).

That said, I don't have a strong opinion here.

-Michael



On Fri, Jul 8, 2016 at 2:55 PM, Matthias J. Sax 
wrote:

> +1
>
> On 07/08/2016 11:03 AM, Eno Thereska wrote:
> > +1 (non-binding)
> >
> >> On 7 Jul 2016, at 18:31, Sriram Subramanian  wrote:
> >>
> >> +1
> >>
> >> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai 
> >> wrote:
> >>
> >>> +1
> >>>
> >>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll 
> wrote:
> >>>
>  +1 (non-binding)
> 
>  On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy 
> >>> wrote:
> 
> > Thanks Henry - we've updated the KIP with an example and the new
> config
> > parameter required. FWIW the user doesn't register a listener, they
>  provide
> > a host:port in config. It is expected they will start a service
> running
>  on
> > that host:port that they can use to connect to the running
> KafkaStreams
> > Instance.
> >
> > Thanks,
> > Damian
> >
> > On Thu, 7 Jul 2016 at 06:06 Henry Cai 
>  wrote:
> >
> >> It wasn't quite clear to me how the user program interacts with the
> >> discovery API, especially on the user supplied listener part, how
> >>> does
> > the
> >> user program supply that listener to KafkaStreams and how does
> > KafkaStreams
> >> know which port the user listener is running, maybe a more complete
> >> end-to-end example including the steps on registering the user
> >>> listener
> > and
> >> whether the user listener needs to be involved with task
> >>> reassignment.
> >>
> >>
> >> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang 
> > wrote:
> >>
> >>> ＋1
> >>>
> >>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy 
> >> wrote:
> >>>
>  Hi all,
> 
>  I'd like to initiate the voting process for KIP-67
>  <
> 
> >>>
> >>
> >
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
> >
> 
>  KAFKA-3909  is
>  the
> >> top
>  level JIRA for this effort.
> 
>  Initial PRs for Step 1 of the process are:
>  Expose State Store Names <
>  https://github.com/apache/kafka/pull/1526>
> >> and
>  Query Local State Stores <
>  https://github.com/apache/kafka/pull/1565>
> 
>  Thanks,
>  Damian
> 
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
> 
> 
> 
>  --
>  Best regards,
>  Michael Noll
> 
> 
> 
>  *Michael G. Noll | Product Manager | Confluent | +1
> 650.453.5860Download
>  Apache Kafka and Confluent Platform: www.confluent.io/download
>  *
> 
> >>>
> >
>
>


-- 
Best regards,
Michael Noll



*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
*

[jira] [Created] (KAFKA-3937) Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

[Tom Crayford (JIRA)]

Tom Crayford created KAFKA-3937:
---

 Summary: Kafka Clients Leak Native Memory For Longer Than Needed 
With Compressed Messages
 Key: KAFKA-3937
 URL: https://issues.apache.org/jira/browse/KAFKA-3937
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.0.0, 0.9.0.1, 0.8.2.2
 Environment: Linux, latest oracle java-8
Reporter: Tom Crayford
Priority: Minor


In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers 
can crash when performing log recovery, as they leak native memory whilst 
decompressing compressed segments, and that native memory isn't cleaned up 
rapidly enough by garbage collection and finalizers. The work to fix that in 
the brokers is taking part in https://github.com/apache/kafka/pull/1598. As 
part of that PR, Ismael Juma asked me to fix similar issues in the client. 
Rather than have one large PR that fixes everything, I'd rather break this work 
up into seperate things, so I'm filing this JIRA to track the followup work. I 
should get to a PR on this at some point relatively soon, once the other PR has 
landed.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367781#comment-15367781
 ] 

Jun Rao commented on KAFKA-3919:


[~BigAndy], yes, the fix could be involved and we haven't nailed down a 
detailed design yet. Zookeeper has to deal with a similar issue. Let me discuss 
this with [~fpj] a bit. Once we are comfortable with a design, we can see if 
you want to pick it up or not.

> Broker faills to start after ungraceful shutdown due to non-monotonically 
> incrementing offsets in logs
> --
>
> Key: KAFKA-3919
> URL: https://issues.apache.org/jira/browse/KAFKA-3919
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1
>Reporter: Andy Coates
>
> Hi All,
> I encountered an issue with Kafka following a power outage that saw a 
> proportion of our cluster disappear. When the power came back on several 
> brokers halted on start up with the error:
> {noformat}
>   Fatal error during KafkaServerStartable startup. Prepare to shutdown”
>   kafka.common.InvalidOffsetException: Attempt to append an offset 
> (1239742691) to position 35728 no larger than the last offset appended 
> (1239742822) to 
> /data3/kafka/mt_xp_its_music_main_itsevent-20/001239444214.index.
>   at 
> kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:207)
>   at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
>   at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at kafka.log.OffsetIndex.append(OffsetIndex.scala:197)
>   at kafka.log.LogSegment.recover(LogSegment.scala:188)
>   at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
>   at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:160)
>   at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>   at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>   at kafka.log.Log.loadSegments(Log.scala:160)
>   at kafka.log.Log.(Log.scala:90)
>   at 
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:150)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The only way to recover the brokers was to delete the log files that 
> contained non monotonically incrementing offsets.
> We've spent some time digging through the logs and I feel I may have worked 
> out the sequence of events leading to this issue, (though this is based on 
> some assumptions I've made about the way Kafka is working, which may be 
> wrong).
> First off, we have unclean leadership elections disable. (We did later enable 
> them to help get around some other issues we were having, but this was 
> several hours after this issue manifested), and we're producing to the topic 
> with gzip compression and acks=1
> We looked through the data logs that were causing the brokers to not start. 
> What we found the initial part of the log has monotonically increasing 
> offset, where each compressed batch normally contained one or two records. 
> Then the is a batch that contains many records, whose first records have an 
> offset below the previous batch and whose last record has an offset above the 
> previous batch. Following on from this there continues a period of large 
> batches, with monotonically increasing offsets, and then the log returns to 
> batches with one or two records.
> Our working assumption here is that the period before the offset dip, with 
> the small batches, is pre-outage normal operation. The period of larger 
> batches is from just after the outage, where producers have a back log to 
> processes when the partition becomes available, and then things return to 
> normal batch sizes again once the back log clears.
> We did also look through the Kafka's application logs to try and piece 
> together the series of events leading up to this. Here’s what we know 
> happened, with regards to one partition that has issues, from the logs:
> Prior to outage:
> * Replicas 

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Damian Guy]

Michael - i'm ok with changing it to a string. Any one else have a strong
opinion on this?

FWIW - i don't think it will work fine as is during the rolling upgrade
scenario as the service that is listening on the port needs to be embedded
within each instance. So for any given instance of a streams application
there will never be both a v1 and v2 alive at the same time (unless of
course the process didn't shutdown properly, but then you have another
problem...).

On Fri, 8 Jul 2016 at 15:26 Michael Noll  wrote:

> I have one further comment about `StreamsConfig.USER_ENDPOINT_CONFIG`.
>
> I think we should consider to not restricting the value of this setting to
> only `host:port` pairs.  By design, this setting is capturing user-driven
> metadata to define an endpoint, so why restrict the creativity or
> flexibility of our users?  I can imagine, for example, that users would
> like to set values such as `https://host:port/api/rest/v1/` in this field
> (e.g. being able to distinguish between `.../v1/` and `.../v2/` may help in
> scenarios such as rolling upgrades, where, during the upgrade, older
> instances may need to coexist with newer instances).
>
> That said, I don't have a strong opinion here.
>
> -Michael
>
>
>
> On Fri, Jul 8, 2016 at 2:55 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 07/08/2016 11:03 AM, Eno Thereska wrote:
> > > +1 (non-binding)
> > >
> > >> On 7 Jul 2016, at 18:31, Sriram Subramanian  wrote:
> > >>
> > >> +1
> > >>
> > >> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai  >
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll 
> > wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy 
> > >>> wrote:
> > 
> > > Thanks Henry - we've updated the KIP with an example and the new
> > config
> > > parameter required. FWIW the user doesn't register a listener, they
> >  provide
> > > a host:port in config. It is expected they will start a service
> > running
> >  on
> > > that host:port that they can use to connect to the running
> > KafkaStreams
> > > Instance.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Thu, 7 Jul 2016 at 06:06 Henry Cai 
> >  wrote:
> > >
> > >> It wasn't quite clear to me how the user program interacts with
> the
> > >> discovery API, especially on the user supplied listener part, how
> > >>> does
> > > the
> > >> user program supply that listener to KafkaStreams and how does
> > > KafkaStreams
> > >> know which port the user listener is running, maybe a more
> complete
> > >> end-to-end example including the steps on registering the user
> > >>> listener
> > > and
> > >> whether the user listener needs to be involved with task
> > >>> reassignment.
> > >>
> > >>
> > >> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang  >
> > > wrote:
> > >>
> > >>> ＋1
> > >>>
> > >>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy <
> damian@gmail.com>
> > >> wrote:
> > >>>
> >  Hi all,
> > 
> >  I'd like to initiate the voting process for KIP-67
> >  <
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
> > >
> > 
> >  KAFKA-3909 
> is
> >  the
> > >> top
> >  level JIRA for this effort.
> > 
> >  Initial PRs for Step 1 of the process are:
> >  Expose State Store Names <
> >  https://github.com/apache/kafka/pull/1526>
> > >> and
> >  Query Local State Stores <
> >  https://github.com/apache/kafka/pull/1565>
> > 
> >  Thanks,
> >  Damian
> > 
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >
> > 
> > 
> > 
> >  --
> >  Best regards,
> >  Michael Noll
> > 
> > 
> > 
> >  *Michael G. Noll | Product Manager | Confluent | +1
> > 650.453.5860Download
> >  Apache Kafka and Confluent Platform: www.confluent.io/download
> >  *
> > 
> > >>>
> > >
> >
> >
>
>
> --
> Best regards,
> Michael Noll
>
>
>
> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
> Apache Kafka and Confluent Platform: www.confluent.io/download
> *
>

[jira] [Commented] (KAFKA-3817) KTableRepartitionMap should handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367817#comment-15367817
 ] 

ASF GitHub Bot commented on KAFKA-3817:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1597


> KTableRepartitionMap should handle null inputs
> --
>
> Key: KAFKA-3817
> URL: https://issues.apache.org/jira/browse/KAFKA-3817
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Jeff Klukas
>Assignee: Guozhang Wang
> Fix For: 0.10.0.1
>
>
> When calling {{KTable.groupBy}} on the result of a KTable-KTable join, NPEs 
> are raised:
> {{org.apache.kafka.streams.kstream.internals.KTableRepartitionMap$
> > KTableMapProcessor.process(KTableRepartitionMap.java:88)}}
> The root cause is that the join is expected to emit null values when no match 
> is found, but KTableRepartitionMap is not set up to handle this case.
> On the users email list, [~guozhang] described a plan of action:
> I think this is actually a bug in KTableRepartitionMap
> that it actually should expect null grouped keys; this would be a
> straight-forward fix for this operator, but I can make a pass over all the
> repartition operators just to make sure they are all gracefully handling
> null keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1597: KAFKA-3817 Follow-up: Avoid forwarding old value i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1597


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3887.
--
   Resolution: Fixed
Fix Version/s: 0.10.0.1

Issue resolved by pull request 1597
[https://github.com/apache/kafka/pull/1597]

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Guozhang Wang
>  Labels: transient-system-test-failure
> Fix For: 0.10.0.1
>
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3708) Rethink exception handling in KafkaStreams

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3708:
-
Issue Type: Sub-task  (was: Bug)
Parent: KAFKA-3938

> Rethink exception handling in KafkaStreams
> --
>
> Key: KAFKA-3708
> URL: https://issues.apache.org/jira/browse/KAFKA-3708
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Guozhang Wang
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> As for 0.10.0.0, the worker threads (i.e. {{StreamThreads}}) can possibly 
> encounter the following runtime exceptions:
> 1) {{consumer.poll()}} could throw KafkaException if some of the 
> configuration are not accepted, such as topics not authorized to read / write 
> (security), session-timeout value not valid, etc; these exceptions will be 
> thrown in the first ever {{poll()}}.
> 2) {{task.addRecords()}} could throw KafkaException (most likely 
> SerializationException) if the deserialization fails.
> 3) {{task.process() / punctuate()}} could throw various KafkaException; for 
> example, serialization / deserialization errors, state storage operation 
> failures (RocksDBException, for example),  producer sending failures, etc.
> 4) {{maybeCommit / commitAll / commitOne}} could throw various Exceptions if 
> the flushing of state store fails, and when {{consumer.commitSync}} throws 
> exceptions other than {{CommitFailedException}}.
> For all the above 4 cases, KafkaStreams does not capture and handle them, but 
> expose them to users, and let users to handle them via 
> {{KafkaStreams.setUncaughtExceptionHandler}}. We need to re-think if the 
> library should just handle these cases without exposing them to users and 
> kill the threads / migrate tasks to others since they are all not recoverable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3752:
-
Issue Type: Sub-task  (was: Improvement)
Parent: KAFKA-3938

> Provide a way for KStreams to recover from unclean shutdown
> ---
>
> Key: KAFKA-3752
> URL: https://issues.apache.org/jira/browse/KAFKA-3752
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>  Labels: architecture
>
> If a KStream application gets killed with SIGKILL (e.g. by the Linux OOM 
> Killer), it may leave behind lock files and fail to recover.
> It would be useful to have an options (say --force) to tell KStreams to 
> proceed even if it finds old LOCK files.
> {noformat}
> [2016-05-24 17:37:52,886] ERROR Failed to create an active task #0_0 in 
> thread [StreamThread-1]:  
> (org.apache.kafka.streams.processor.internals.StreamThread:583)
> org.apache.kafka.streams.errors.ProcessorStateException: Error while creating 
> the state manager
>   at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamTh

[jira] [Updated] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3758:
-
Issue Type: Sub-task  (was: Bug)
Parent: KAFKA-3938

> KStream job fails to recover after Kafka broker stopped
> ---
>
> Key: KAFKA-3758
> URL: https://issues.apache.org/jira/browse/KAFKA-3758
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Assignee: Guozhang Wang
> Attachments: muon.log.1.gz
>
>
> We've been doing some testing of a fairly complex KStreams job and under load 
> it seems the job fails to rebalance + recover if we shut down one of the 
> kafka brokers. The test we were running had a 3-node kafka cluster where each 
> topic had at least a replication factor of 2, and we terminated one of the 
> nodes.
> Attached is the full log, the root exception seems to be contention on the 
> lock on the state directory. The job continues to try to recover but throws 
> errors relating to locks over and over. Restarting the job itself resolves 
> the problem.
>  1702 org.apache.kafka.streams.errors.ProcessorStateException: Error while 
> creating the state manager
>  1703 at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
>  1704 at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
>  1705 at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
>  1706 at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
>  1707 at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
>  1708 at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
>  1709 at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
>  1710 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
>  1711 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
>  1712 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1713 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1714 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
>  1715 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1716 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1717 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
>  1718 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
>  1719 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
>  1720 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
>  1721 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
>  1722 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>  1723 at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>  1724 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
>  1725 at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>  1726 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>  1727 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>  1728 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>  1729 at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>  1730 at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
>  1731 at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(Consu

[jira] [Created] (KAFKA-3938) Fix consumer session timeout issue in Kafka Streams

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3938:


 Summary: Fix consumer session timeout issue in Kafka Streams
 Key: KAFKA-3938
 URL: https://issues.apache.org/jira/browse/KAFKA-3938
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


After KIP-62 / KAFKA-3888 is merged in, Kafka Streams should leverage this new 
feature to fix the session timeout issue that can be caused by:

1. long rebalance period due to state store restoration.
2. exceptional long processing time for a batch of records.

Also we need to consider:

1. state store directory locking mechanism between rebalances while one 
instance is grabbing tasks from another instance on the same machine.
2. exceptional handling in Kafka Streams: what should we really expose to users 
and what should be handled automatically?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3938) Fix consumer session timeout issue in Kafka Streams

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3938:
-
Labels: architecture  (was: )

> Fix consumer session timeout issue in Kafka Streams
> ---
>
> Key: KAFKA-3938
> URL: https://issues.apache.org/jira/browse/KAFKA-3938
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>  Labels: architecture
>
> After KIP-62 / KAFKA-3888 is merged in, Kafka Streams should leverage this 
> new feature to fix the session timeout issue that can be caused by:
> 1. long rebalance period due to state store restoration.
> 2. exceptional long processing time for a batch of records.
> Also we need to consider:
> 1. state store directory locking mechanism between rebalances while one 
> instance is grabbing tasks from another instance on the same machine.
> 2. exceptional handling in Kafka Streams: what should we really expose to 
> users and what should be handled automatically?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3559) Task creation time taking too long in rebalance callback

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3559:
-
Issue Type: Sub-task  (was: Bug)
Parent: KAFKA-3938

> Task creation time taking too long in rebalance callback
> 
>
> Key: KAFKA-3559
> URL: https://issues.apache.org/jira/browse/KAFKA-3559
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: architecture
> Fix For: 0.10.1.0
>
>
> Currently in Kafka Streams, we create stream tasks upon getting newly 
> assigned partitions in rebalance callback function {code} onPartitionAssigned 
> {code}, which involves initialization of the processor state stores as well 
> (including opening the rocksDB, restore the store from changelog, etc, which 
> takes time).
> With a large number of state stores, the initialization time itself could 
> take tens of seconds, which usually is larger than the consumer session 
> timeout. As a result, when the callback is completed, the consumer is already 
> treated as failed by the coordinator and rebalance again.
> We need to consider if we can optimize the initialization process, or move it 
> out of the callback function, and while initializing the stores one-by-one, 
> use poll call to send heartbeats to avoid being kicked out by coordinator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1408

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is

--
[...truncated 3325 lines...]

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testOffsetAssignmentAfterMessageFormatConversion STARTED

kafka.message.ByteBufferMessageSetTest > 
testOffsetAssignmentAfterMessageFormatConversion PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testAbsoluteOffsetAssignment STARTED

kafka.message.ByteBufferMessageSetTest > testAbsoluteOffsetAssignment PASSED

kafka.message.ByteBufferMessageSetTest > testCreateTime STARTED

kafka.message.ByteBufferMessageSetTest > testCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testInvalidCreateTime STARTED

kafka.message.ByteBufferMessageSetTest > testInvalidCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testLogAppendTime STARTED

kafka.message.ByteBufferMessageSetTest > testLogAppendTime PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo STARTED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testRelativeOffsetAssignment STARTED

kafka.message.ByteBufferMessageSetTest > testRelativeOffsetAssignment PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown STARTED

kafka.netwo

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367923#comment-15367923
 ] 

Bill Bejeck commented on KAFKA-3101:


[~guozhang] [~enothereska] 

Would adding flatbuffers (https://google.github.io/flatbuffers/) be beyond the 
scope of this performance comparison?

> Optimize Aggregation Outputs
> 
>
> Key: KAFKA-3101
> URL: https://issues.apache.org/jira/browse/KAFKA-3101
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
> Fix For: 0.10.1.0
>
>
> Today we emit one output record for each incoming message for Table / 
> Windowed Stream Aggregations. For example, say we have a sequence of 
> aggregate outputs computed from the input stream (assuming there is no agg 
> value for this key before):
> V1, V2, V3, V4, V5
> Then the aggregator will output the following sequence of Change oldValue>:
> , , , , 
> where could cost a lot of CPU overhead computing the intermediate results. 
> Instead if we can let the underlying state store to "remember" the last 
> emitted old value, we can reduce the number of emits based on some configs. 
> More specifically, we can add one more field in the KV store engine storing 
> the last emitted old value, which only get updated when we emit to the 
> downstream processor. For example:
> At Beginning: 
> Store: key => empty (no agg values yet)
> V1 computed: 
> Update Both in Store: key => (V1, V1), Emit 
> V2 computed: 
> Update NewValue in Store: key => (V2, V1), No Emit
> V3 computed: 
> Update NewValue in Store: key => (V3, V1), No Emit
> V4 computed: 
> Update Both in Store: key => (V4, V4), Emit 
> V5 computed: 
> Update NewValue in Store: key => (V5, V4), No Emit
> One more thing to consider is that, we need a "closing" time control on the 
> not-yet-emitted keys; when some time has elapsed (or the window is to be 
> closed), we need to check for any key if their current materialized pairs 
> have not been emitted (for example  in the above example). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3939) add new consumer metrics in docs

[Jun Rao (JIRA)]

Jun Rao created KAFKA-3939:
--

 Summary: add new consumer metrics in docs
 Key: KAFKA-3939
 URL: https://issues.apache.org/jira/browse/KAFKA-3939
 Project: Kafka
  Issue Type: Task
  Components: consumer
Affects Versions: 0.10.0.0
Reporter: Jun Rao


In the monitoring section of our documentation, we have metrics for the broker 
and the producer. It would be useful to add the metrics for the new java 
consumer as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Michael Noll]

Addendum in case my previous email wasn't clear:

> So for any given instance of a streams application there will never be
both a v1 and v2 alive at the same time

That's right.  But the current live instance will be able to tell other
instances, via its endpoint setting, whether it wants to be contacted at v1
or at v2.  The other instances can't guess that.  Think: if an older
instance would manually compose the "rest" of an endpoint URI, having only
the host and port from the endpoint setting, it might not know that the new
instances have a different endpoint suffix, for example).


On Fri, Jul 8, 2016 at 6:37 PM, Michael Noll  wrote:

> Damian,
>
> about the rolling upgrade comment:  An instance A will contact another
> instance B by the latter's endpoint, right?  So if A has no further
> information available than B's host and port, then how should instance A
> know whether it should call B at /v1/ or at /v2/?  I agree that my
> suggestion isn't foolproof, but it is afaict better than the host:port
> approach.
>
>
>
> On Fri, Jul 8, 2016 at 5:15 PM, Damian Guy  wrote:
>
>> Michael - i'm ok with changing it to a string. Any one else have a strong
>> opinion on this?
>>
>> FWIW - i don't think it will work fine as is during the rolling upgrade
>> scenario as the service that is listening on the port needs to be embedded
>> within each instance. So for any given instance of a streams application
>> there will never be both a v1 and v2 alive at the same time (unless of
>> course the process didn't shutdown properly, but then you have another
>> problem...).
>>
>> On Fri, 8 Jul 2016 at 15:26 Michael Noll  wrote:
>>
>> > I have one further comment about `StreamsConfig.USER_ENDPOINT_CONFIG`.
>> >
>> > I think we should consider to not restricting the value of this setting
>> to
>> > only `host:port` pairs.  By design, this setting is capturing
>> user-driven
>> > metadata to define an endpoint, so why restrict the creativity or
>> > flexibility of our users?  I can imagine, for example, that users would
>> > like to set values such as `https://host:port/api/rest/v1/` in this
>> field
>> > (e.g. being able to distinguish between `.../v1/` and `.../v2/` may
>> help in
>> > scenarios such as rolling upgrades, where, during the upgrade, older
>> > instances may need to coexist with newer instances).
>> >
>> > That said, I don't have a strong opinion here.
>> >
>> > -Michael
>> >
>> >
>> >
>> > On Fri, Jul 8, 2016 at 2:55 PM, Matthias J. Sax 
>> > wrote:
>> >
>> > > +1
>> > >
>> > > On 07/08/2016 11:03 AM, Eno Thereska wrote:
>> > > > +1 (non-binding)
>> > > >
>> > > >> On 7 Jul 2016, at 18:31, Sriram Subramanian 
>> wrote:
>> > > >>
>> > > >> +1
>> > > >>
>> > > >> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai
>> > > >
>> > > >> wrote:
>> > > >>
>> > > >>> +1
>> > > >>>
>> > > >>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll <
>> mich...@confluent.io>
>> > > wrote:
>> > > >>>
>> > >  +1 (non-binding)
>> > > 
>> > >  On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy <
>> damian@gmail.com>
>> > > >>> wrote:
>> > > 
>> > > > Thanks Henry - we've updated the KIP with an example and the new
>> > > config
>> > > > parameter required. FWIW the user doesn't register a listener,
>> they
>> > >  provide
>> > > > a host:port in config. It is expected they will start a service
>> > > running
>> > >  on
>> > > > that host:port that they can use to connect to the running
>> > > KafkaStreams
>> > > > Instance.
>> > > >
>> > > > Thanks,
>> > > > Damian
>> > > >
>> > > > On Thu, 7 Jul 2016 at 06:06 Henry Cai
>> 
>> > >  wrote:
>> > > >
>> > > >> It wasn't quite clear to me how the user program interacts with
>> > the
>> > > >> discovery API, especially on the user supplied listener part,
>> how
>> > > >>> does
>> > > > the
>> > > >> user program supply that listener to KafkaStreams and how does
>> > > > KafkaStreams
>> > > >> know which port the user listener is running, maybe a more
>> > complete
>> > > >> end-to-end example including the steps on registering the user
>> > > >>> listener
>> > > > and
>> > > >> whether the user listener needs to be involved with task
>> > > >>> reassignment.
>> > > >>
>> > > >>
>> > > >> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang <
>> wangg...@gmail.com
>> > >
>> > > > wrote:
>> > > >>
>> > > >>> ＋1
>> > > >>>
>> > > >>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy <
>> > damian@gmail.com>
>> > > >> wrote:
>> > > >>>
>> > >  Hi all,
>> > > 
>> > >  I'd like to initiate the voting process for KIP-67
>> > >  <
>> > > 
>> > > >>>
>> > > >>
>> > > >
>> > > 
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
>> > > >
>> > > 
>> > >  KAFKA-3909 > >
>> > is
>> >

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Michael Noll]

Damian,

about the rolling upgrade comment:  An instance A will contact another
instance B by the latter's endpoint, right?  So if A has no further
information available than B's host and port, then how should instance A
know whether it should call B at /v1/ or at /v2/?  I agree that my
suggestion isn't foolproof, but it is afaict better than the host:port
approach.



On Fri, Jul 8, 2016 at 5:15 PM, Damian Guy  wrote:

> Michael - i'm ok with changing it to a string. Any one else have a strong
> opinion on this?
>
> FWIW - i don't think it will work fine as is during the rolling upgrade
> scenario as the service that is listening on the port needs to be embedded
> within each instance. So for any given instance of a streams application
> there will never be both a v1 and v2 alive at the same time (unless of
> course the process didn't shutdown properly, but then you have another
> problem...).
>
> On Fri, 8 Jul 2016 at 15:26 Michael Noll  wrote:
>
> > I have one further comment about `StreamsConfig.USER_ENDPOINT_CONFIG`.
> >
> > I think we should consider to not restricting the value of this setting
> to
> > only `host:port` pairs.  By design, this setting is capturing user-driven
> > metadata to define an endpoint, so why restrict the creativity or
> > flexibility of our users?  I can imagine, for example, that users would
> > like to set values such as `https://host:port/api/rest/v1/` in this
> field
> > (e.g. being able to distinguish between `.../v1/` and `.../v2/` may help
> in
> > scenarios such as rolling upgrades, where, during the upgrade, older
> > instances may need to coexist with newer instances).
> >
> > That said, I don't have a strong opinion here.
> >
> > -Michael
> >
> >
> >
> > On Fri, Jul 8, 2016 at 2:55 PM, Matthias J. Sax 
> > wrote:
> >
> > > +1
> > >
> > > On 07/08/2016 11:03 AM, Eno Thereska wrote:
> > > > +1 (non-binding)
> > > >
> > > >> On 7 Jul 2016, at 18:31, Sriram Subramanian 
> wrote:
> > > >>
> > > >> +1
> > > >>
> > > >> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai
>  > >
> > > >> wrote:
> > > >>
> > > >>> +1
> > > >>>
> > > >>> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll  >
> > > wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy  >
> > > >>> wrote:
> > > 
> > > > Thanks Henry - we've updated the KIP with an example and the new
> > > config
> > > > parameter required. FWIW the user doesn't register a listener,
> they
> > >  provide
> > > > a host:port in config. It is expected they will start a service
> > > running
> > >  on
> > > > that host:port that they can use to connect to the running
> > > KafkaStreams
> > > > Instance.
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > > On Thu, 7 Jul 2016 at 06:06 Henry Cai  >
> > >  wrote:
> > > >
> > > >> It wasn't quite clear to me how the user program interacts with
> > the
> > > >> discovery API, especially on the user supplied listener part,
> how
> > > >>> does
> > > > the
> > > >> user program supply that listener to KafkaStreams and how does
> > > > KafkaStreams
> > > >> know which port the user listener is running, maybe a more
> > complete
> > > >> end-to-end example including the steps on registering the user
> > > >>> listener
> > > > and
> > > >> whether the user listener needs to be involved with task
> > > >>> reassignment.
> > > >>
> > > >>
> > > >> On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > wrote:
> > > >>
> > > >>> ＋1
> > > >>>
> > > >>> On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy <
> > damian@gmail.com>
> > > >> wrote:
> > > >>>
> > >  Hi all,
> > > 
> > >  I'd like to initiate the voting process for KIP-67
> > >  <
> > > 
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
> > > >
> > > 
> > >  KAFKA-3909 
> > is
> > >  the
> > > >> top
> > >  level JIRA for this effort.
> > > 
> > >  Initial PRs for Step 1 of the process are:
> > >  Expose State Store Names <
> > >  https://github.com/apache/kafka/pull/1526>
> > > >> and
> > >  Query Local State Stores <
> > >  https://github.com/apache/kafka/pull/1565>
> > > 
> > >  Thanks,
> > >  Damian
> > > 
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> -- Guozhang
> > > >>>
> > > >>
> > > >
> > > 
> > > 
> > > 
> > >  --
> > >  Best regards,
> > >  Michael Noll
> > > 
> > > 
> > > 
> > >  *Michael G. Noll | Product Manager | Confluent | +1
> > > 650.453.5860Download
> > >  Apache Kafka and Confluent Platform: www.confluent.io/download
>

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367982#comment-15367982
 ] 

Eno Thereska commented on KAFKA-3101:
-

[~bbejeck] Could you provide a bit more context on why flatbuffers are needed 
for the comparison? Thanks.

> Optimize Aggregation Outputs
> 
>
> Key: KAFKA-3101
> URL: https://issues.apache.org/jira/browse/KAFKA-3101
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
> Fix For: 0.10.1.0
>
>
> Today we emit one output record for each incoming message for Table / 
> Windowed Stream Aggregations. For example, say we have a sequence of 
> aggregate outputs computed from the input stream (assuming there is no agg 
> value for this key before):
> V1, V2, V3, V4, V5
> Then the aggregator will output the following sequence of Change oldValue>:
> , , , , 
> where could cost a lot of CPU overhead computing the intermediate results. 
> Instead if we can let the underlying state store to "remember" the last 
> emitted old value, we can reduce the number of emits based on some configs. 
> More specifically, we can add one more field in the KV store engine storing 
> the last emitted old value, which only get updated when we emit to the 
> downstream processor. For example:
> At Beginning: 
> Store: key => empty (no agg values yet)
> V1 computed: 
> Update Both in Store: key => (V1, V1), Emit 
> V2 computed: 
> Update NewValue in Store: key => (V2, V1), No Emit
> V3 computed: 
> Update NewValue in Store: key => (V3, V1), No Emit
> V4 computed: 
> Update Both in Store: key => (V4, V4), Emit 
> V5 computed: 
> Update NewValue in Store: key => (V5, V4), No Emit
> One more thing to consider is that, we need a "closing" time control on the 
> not-yet-emitted keys; when some time has elapsed (or the window is to be 
> closed), we need to check for any key if their current materialized pairs 
> have not been emitted (for example  in the above example). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.0-jdk7 #145

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is

--
[...truncated 1670 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsWrongSetValue PASSED

kafka.KafkaTest > testKafkaSslPasswords PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgs PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheEnd PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsOnly PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheBegging PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testLZ4FramingV0 PASSED

kafka.message.MessageCompressionTest > testLZ4FramingV1 PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageWriterTest > testWithNoCompressionAttribute PASSED

kafka.message.MessageWriterTest > testWithCompressionAttribute PASSED

kafka.message.MessageWriterTest > testBufferingOutputStream PASSED

kafka.message.MessageWriterTest > testWithKey PASSED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.message.MessageTest > testMessageFormatConversion PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testOffsetAssignmentAfterMessageFormatConversion PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testAbsoluteOffsetAssignment PASSED

kafka.message.ByteBufferMessageSetTest > testCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testInvalidCreateTime PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testLogAppendTime PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testRelativeOffsetAssignment PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PAS

Build failed in Jenkins: kafka-trunk-jdk8 #739

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is 
null

--
[...truncated 3356 lines...]

kafka.coordinator.GroupMetadataManagerTest > testStoreNonEmptyGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols STARTED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols PASSED

kafka.coordinator.MemberMetadataTest > testMetadata STARTED

kafka.coordinator.MemberMetadataTest > testMetadata PASSED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
STARTED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
PASSED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol STARTED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol PASSED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
STARTED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
PASSED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure PASSED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocol STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocol PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
PASSED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync STARTED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync PASSED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers PASSED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToDeadTransition 
STARTED

kafka

[GitHub] kafka pull request #1574: KAFKA-3920: Add Schema source connector to Kafka C...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1574


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3920) Add Schema source connector to Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368075#comment-15368075
 ] 

ASF GitHub Bot commented on KAFKA-3920:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1574


> Add Schema source connector to Kafka Connect
> 
>
> Key: KAFKA-3920
> URL: https://issues.apache.org/jira/browse/KAFKA-3920
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
> Fix For: 0.10.1.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Many sink connectors needs to validate schema compatibility during system 
> tests and thus needs a source connector that provides capability to send data 
> to Kafka with multiple schemas.
> The schema source connector is also useful to validate delivery guarantees.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3920) Add Schema source connector to Kafka Connect

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3920.
--
Resolution: Fixed

Issue resolved by pull request 1574
[https://github.com/apache/kafka/pull/1574]

> Add Schema source connector to Kafka Connect
> 
>
> Key: KAFKA-3920
> URL: https://issues.apache.org/jira/browse/KAFKA-3920
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Liquan Pei
>Assignee: Liquan Pei
> Fix For: 0.10.1.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Many sink connectors needs to validate schema compatibility during system 
> tests and thus needs a source connector that provides capability to send data 
> to Kafka with multiple schemas.
> The schema source connector is also useful to validate delivery guarantees.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1409

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3920: Add Schema source connector to Kafka Connect

--
[...truncated 5676 lines...]

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild STARTED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls STARTED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck STARTED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.log.LogTest > testAsyncDelete STARTED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange STARTED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException STARTED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException PASSED

kafka.log.LogTest > testReadAtLogGap STARTED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll STARTED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog STARTED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck STARTED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation STARTED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints STARTED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset STARTED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testDeleteOldSegmentsMethod STARTED

kafka.log.LogTest > testDeleteOldSegmentsMethod PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull STARTED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild STARTED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved STARTED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages STARTED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload STARTED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog STARTED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset STARTED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate STARTED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName STARTED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles STARTED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll STARTED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter STARTED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName STARTED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo STARTED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile STARTED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.LogConfigTest > testFromPropsEmpty STARTED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps STARTED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid STARTED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.CleanerTest > testBuildOffsetMap STARTED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testBuildOffsetMapFa

[jira] [Created] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Jun Rao (JIRA)]

Jun Rao created KAFKA-3940:
--

 Summary: Log should check the return value of dir.mkdirs()
 Key: KAFKA-3940
 URL: https://issues.apache.org/jira/browse/KAFKA-3940
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.10.0.0
Reporter: Jun Rao


In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
just assume the directory will exist after the call. However, if the directory 
can't be created (e.g. due to no space), we will hit NullPointerException in 
the next statement, which will be confusing.

   for(file <- dir.listFiles if file.isFile) {




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-3940:
---
Labels: newbie  (was: )

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368315#comment-15368315
 ] 

Jun Rao commented on KAFKA-3940:


Also, instead of using File.mkdirs(), it may be better to use 
Files.createDirectory() instead. The latter throws IOException on error instead 
of returning false and will potentially give better indication on the cause.

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-3940:
-

Assignee: Ishita Mandhan

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #740

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-3939) add new consumer metrics in docs

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368391#comment-15368391
 ] 

James Cheng commented on KAFKA-3939:


I am addressing this work in https://issues.apache.org/jira/browse/KAFKA-3480. 
There is a patch available. Can you help me get someone to review it?

> add new consumer metrics in docs
> 
>
> Key: KAFKA-3939
> URL: https://issues.apache.org/jira/browse/KAFKA-3939
> Project: Kafka
>  Issue Type: Task
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>
> In the monitoring section of our documentation, we have metrics for the 
> broker and the producer. It would be useful to add the metrics for the new 
> java consumer as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3163) KIP-33 - Add a time based log index to Kafka

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-3163:
---
Attachment: 00113931.timeindex
00113931.log

Also, I ran the following tool in the patch on the attached timeindex file and 
got a bunch of errors. 

kafka-run-class.sh kafka.tools.DumpLogSegments --files 
/tmp/kafka-logs/test-0/00113931.timeindex
timestamp: 1468010788307(log timestamp: 1468010788303) offset: 113946(log 
offset: 113931)
timestamp: 1468010788308(log timestamp: 1468010788303) offset: 113976(log 
offset: 113931)
timestamp: 1468010788309(log timestamp: 1468010788303) offset: 114006(log 
offset: 113931)
timestamp: 1468010788310(log timestamp: 1468010788303) offset: 114036(log 
offset: 113931)
timestamp: 1468010788311(log timestamp: 1468010788303) offset: 114081(log 
offset: 113931)
timestamp: 1468010788312(log timestamp: 1468010788303) offset: 114111(log 
offset: 113931)
timestamp: 1468010788313(log timestamp: 1468010788303) offset: 114156(log 
offset: 113931)
timestamp: 1468010788314(log timestamp: 1468010788303) offset: 114201(log 
offset: 113931)
Found timestamp mismatch in 
:/tmp/kafka-logs/test-0/00113931.timeindex
  Index timestamp: 1468010788314, log timestamp: 1468010788303
  Index timestamp: 1468010788313, log timestamp: 1468010788303
  Index timestamp: 1468010788312, log timestamp: 1468010788303
  Index timestamp: 1468010788311, log timestamp: 1468010788303
  Index timestamp: 1468010788310, log timestamp: 1468010788303
  Index timestamp: 1468010788309, log timestamp: 1468010788303
  Index timestamp: 1468010788308, log timestamp: 1468010788303
  Index timestamp: 1468010788307, log timestamp: 1468010788303

A few things: (1) The data is populated with the producer performance tool. So, 
the time index shouldn't be corrupted. (2) It's a bit hard to understand the 
output. It's not clear what's in the bracket really means and it's not clear 
that indicates an error. It's also a bit weird that the same error is print 
again later and in reverse timestamp order.

> KIP-33 - Add a time based log index to Kafka
> 
>
> Key: KAFKA-3163
> URL: https://issues.apache.org/jira/browse/KAFKA-3163
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
> Attachments: 00113931.log, 00113931.timeindex
>
>
> This ticket is associated with KIP-33.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1599: KAFKA-2941: Clarify docs for key and value Convert...

[ewencp]

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/1599

KAFKA-2941: Clarify docs for key and value Converters



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2941-explain-converter-configs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1599


commit e45de3a0d41f3ef4f31c32fed1bc56cc9b4281bf
Author: Ewen Cheslack-Postava 
Date:   2016-07-08T21:52:07Z

KAFKA-2941: Clarify docs for key and value Converters




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2941) Docs for key/value converter in Kafka connect are unclear

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368573#comment-15368573
 ] 

ASF GitHub Bot commented on KAFKA-2941:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/1599

KAFKA-2941: Clarify docs for key and value Converters



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2941-explain-converter-configs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1599


commit e45de3a0d41f3ef4f31c32fed1bc56cc9b4281bf
Author: Ewen Cheslack-Postava 
Date:   2016-07-08T21:52:07Z

KAFKA-2941: Clarify docs for key and value Converters




> Docs for key/value converter in Kafka connect are unclear
> -
>
> Key: KAFKA-2941
> URL: https://issues.apache.org/jira/browse/KAFKA-2941
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> These docs don't really explain what the configs do or why users might want 
> to change them.
> Via [~gwenshap], something like this would be better: "Converter class for 
> key Connect data. This controls the format of the data that will be written 
> either to Kafka or to a sink system. Popular formats include Json and Avro"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2941) Docs for key/value converter in Kafka connect are unclear

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-2941:
-
Status: Patch Available  (was: Open)

> Docs for key/value converter in Kafka connect are unclear
> -
>
> Key: KAFKA-2941
> URL: https://issues.apache.org/jira/browse/KAFKA-2941
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> These docs don't really explain what the configs do or why users might want 
> to change them.
> Via [~gwenshap], something like this would be better: "Converter class for 
> key Connect data. This controls the format of the data that will be written 
> either to Kafka or to a sink system. Popular formats include Json and Avro"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: SSL: Broker works but consumer/producer fail

[Harsha Chintalapani]

Which version of kafka are you using.
-Harsha

On Fri, Jul 8, 2016 at 1:09 AM Narendra Bidari 
wrote:

> Hi Vineet,
>
> The setup of ssl Kafka requires to make one too many steps precise
> correct. I have listed some below . Hope it helps
> https://github.com/Symantec/kafka-security-0.9
>
> Sent from my iPhone
>
> Regards
>
> > On Jul 7, 2016, at 4:49 PM, Vineet Kumar  wrote:
> >
> > Hi
> >  I followed Apache Kafka SSL instructions verbatim but my producer and
> > consumer both hang or error out as follows.
> > openssl s_client BTW does work fine with the server below yielding
> > certificates etc thereby confirming that the server can talk back SSL.
> >
> >
> > *Producer and Consumer*
> > =
> >
> > Config changes (client-ssl.properties)
> > ---
> > security.protocol=SSL
> >
> > % bin/kafka-console-*consumer*.sh --bootstrap-server 192.168.1.XXX:9093
> > --topic test --new-consumer --consumer.config
> config/client-ssl.properties
> > **
> >
> > % bin/kafka-console-*producer*.sh --broker-list 192.168.1.XXX:9093
> --topic
> > test --producer.config config/client-ssl.properties
> > a
> >
> > **
> >
> > [2016-07-07 16:35:57,670] ERROR Error when sending message to topic test
> > with key: null, value: 29 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.TimeoutException: Failed to update
> metadata
> > after 6 ms.
> >
> > *Broker*
> > ==
> > Config changes (server.properties)
> > ---
> > listeners=SSL://192.168.1.XXX:9093
> > security.inter.broker.protocol=SSL
> > advertised.listeners=SSL://192.168.1.XXX:9093
> > ssl.keystore.location=/<..>/server.keystore.jks
> > ssl.keystore.password=
> > ssl.key.password=
> >
> > % bin/kafka-*server*-start.sh config/server.properties
> > [2016-07-07 16:14:00,805] INFO Registered broker 0 at path /brokers/ids/0
> > with addresses: *SSL -> EndPoint(192.168.1.XXX,9093,SSL)*
> > (kafka.utils.ZkUtils)
> > [2016-07-07 16:14:00,820] INFO New leader is 0
> > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> > [2016-07-07 16:14:00,825] INFO Kafka version : 0.10.0.0
> > (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-07-07 16:14:00,825] INFO Kafka commitId : b8642491e78c5a13
> > (org.apache.kafka.common.utils.AppInfoParser)
> > [2016-07-07 16:14:00,827] INFO [Kafka Server 0], started
> > (kafka.server.KafkaServer)
> >
> > *Zookeeper*
> > =
> > Config changes
> > ---
> >  Nothing
> >
> > % bin/zookeeper-server-start.sh config/zookeeper.properties
> > 
> > 
> > [2016-07-07 16:13:18,002] INFO binding to port 0.0.0.0/0.0.0.0:2181
> > (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > 
> > 
> > [2016-07-07 16:14:00,131] INFO Accepted socket connection from /
> > 127.0.0.1:41188 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> > [2016-07-07 16:14:00,189] INFO Client attempting to establish new session
> > at /127.0.0.1:41188 (org.apache.zookeeper.server.ZooKeeperServer)
> > [2016-07-07 16:14:00,199] INFO Established session 0x155c7a306dc with
> > negotiated timeout 6000 for client /127.0.0.1:41188
> > (org.apache.zookeeper.server.ZooKeeperServer)
> > [2016-07-07 16:14:00,652] INFO Got user-level KeeperException when
> > processing sessionid:0x155c7a306dc type:delete cxid:0x22 zxid:0xd6
> > txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election
> > Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election
> > (org.apache.zookeeper.server.PrepRequestProcessor)
> > [2016-07-07 16:14:00,778] INFO Got user-level KeeperException when
> > processing sessionid:0x155c7a306dc type:create cxid:0x29 zxid:0xd7
> > txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode =
> > NodeExists for /brokers
> (org.apache.zookeeper.server.PrepRequestProcessor)
> > [2016-07-07 16:14:00,778] INFO Got user-level KeeperException when
> > processing sessionid:0x155c7a306dc type:create cxid:0x2a zxid:0xd8
> > txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode =
> > NodeExists for /brokers/ids
> > (org.apache.zookeeper.server.PrepRequestProcessor)
>

[jira] [Updated] (KAFKA-3111) java.lang.ArithmeticException: / by zero in ConsumerPerformance

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-3111:
---
   Resolution: Fixed
Fix Version/s: 0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 788
[https://github.com/apache/kafka/pull/788]

> java.lang.ArithmeticException: / by zero in ConsumerPerformance
> ---
>
> Key: KAFKA-3111
> URL: https://issues.apache.org/jira/browse/KAFKA-3111
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> Saw the following error. If there are lots of unconsumed messages, between 
> two iterations of the consumption, the timestamp may not have changed.
> kafka-consumer-perf-test --zookeeper localhost:2181 --topic test --group 
> test-group --threads 1 --show-detailed-stats --reporting-interval 5000
> 2016-01-13 09:12:43:905, 0, 1048576, 35.2856, 238.4186, 37, 250.
> 2016-01-13 09:12:43:916, 0, 1048576, 35.7624, 47.6837, 375000, 50.
> java.lang.ArithmeticException: / by zero
> at 
> kafka.tools.ConsumerPerformance$ConsumerPerfThread.printMessage(ConsumerPerformance.scala:189)
> at 
> kafka.tools.ConsumerPerformance$ConsumerPerfThread.run(ConsumerPerformance.scala:164)
> 2016-01-13 09:12:43:918, 0, 1048576, 36.2393, 0.7117, 38, 7000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3111) java.lang.ArithmeticException: / by zero in ConsumerPerformance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368670#comment-15368670
 ] 

ASF GitHub Bot commented on KAFKA-3111:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/788


> java.lang.ArithmeticException: / by zero in ConsumerPerformance
> ---
>
> Key: KAFKA-3111
> URL: https://issues.apache.org/jira/browse/KAFKA-3111
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
> Fix For: 0.10.1.0
>
>
> Saw the following error. If there are lots of unconsumed messages, between 
> two iterations of the consumption, the timestamp may not have changed.
> kafka-consumer-perf-test --zookeeper localhost:2181 --topic test --group 
> test-group --threads 1 --show-detailed-stats --reporting-interval 5000
> 2016-01-13 09:12:43:905, 0, 1048576, 35.2856, 238.4186, 37, 250.
> 2016-01-13 09:12:43:916, 0, 1048576, 35.7624, 47.6837, 375000, 50.
> java.lang.ArithmeticException: / by zero
> at 
> kafka.tools.ConsumerPerformance$ConsumerPerfThread.printMessage(ConsumerPerformance.scala:189)
> at 
> kafka.tools.ConsumerPerformance$ConsumerPerfThread.run(ConsumerPerformance.scala:164)
> 2016-01-13 09:12:43:918, 0, 1048576, 36.2393, 0.7117, 38, 7000.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #788: KAFKA-3111: Fix ConsumerPerformance reporting to us...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/788


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368797#comment-15368797
 ] 

Ismael Juma commented on KAFKA-3940:


I think we should just change all usages of File.mkdirs() to 
Files.createDirectory(). And similarly for `File.delete` versus `Files.delete`.

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reopened KAFKA-3887:
--

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Guozhang Wang
>  Labels: transient-system-test-failure
> Fix For: 0.10.0.1
>
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368817#comment-15368817
 ] 

Guozhang Wang commented on KAFKA-3887:
--

The system test still failing but seems in a different way: 
https://jenkins.confluent.io/job/system-test-kafka/275/consoleFull

Reopen this issue.

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Guozhang Wang
>  Labels: transient-system-test-failure
> Fix For: 0.10.0.1
>
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1410

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-3111; Fix ConsumerPerformance reporting to use time-based instead

--
[...truncated 9017 lines...]

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild STARTED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls STARTED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck STARTED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.log.LogTest > testAsyncDelete STARTED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange STARTED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException STARTED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException PASSED

kafka.log.LogTest > testReadAtLogGap STARTED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll STARTED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog STARTED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck STARTED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation STARTED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints STARTED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset STARTED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testDeleteOldSegmentsMethod STARTED

kafka.log.LogTest > testDeleteOldSegmentsMethod PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull STARTED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild STARTED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved STARTED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages STARTED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload STARTED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog STARTED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset STARTED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate STARTED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName STARTED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles STARTED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll STARTED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter STARTED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName STARTED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo STARTED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile STARTED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.LogConfigTest > testFromPropsEmpty STARTED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps STARTED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid STARTED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.CleanerTest > testBuildOffsetMap STARTED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > t

Build failed in Jenkins: kafka-trunk-jdk8 #741

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-3111; Fix ConsumerPerformance reporting to use time-based instead

--
[...truncated 5610 lines...]

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild STARTED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls STARTED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck STARTED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.log.LogTest > testAsyncDelete STARTED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange STARTED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException STARTED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException PASSED

kafka.log.LogTest > testReadAtLogGap STARTED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll STARTED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog STARTED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck STARTED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation STARTED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints STARTED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset STARTED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testDeleteOldSegmentsMethod STARTED

kafka.log.LogTest > testDeleteOldSegmentsMethod PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull STARTED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild STARTED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved STARTED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages STARTED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload STARTED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog STARTED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset STARTED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate STARTED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName STARTED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles STARTED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll STARTED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter STARTED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName STARTED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo STARTED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile STARTED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.LogConfigTest > testFromPropsEmpty STARTED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps STARTED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid STARTED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.CleanerTest > testBuildOffsetMap STARTED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > te

[jira] [Created] (KAFKA-3941) Avoid applying eviction listener in InMemoryKeyValueLoggedStore

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3941:


 Summary: Avoid applying eviction listener in 
InMemoryKeyValueLoggedStore
 Key: KAFKA-3941
 URL: https://issues.apache.org/jira/browse/KAFKA-3941
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang
Assignee: Guozhang Wang


This is reported by [~norwood].

In {{InMemoryKeyValueLoggedStore}} we set the eviction listener which records 
the evicted records as deletes in the changelog. However, when restoring the 
store this listener will then double-writes the delete record, causing the 
restoration process to fail.

We should defer the listener initialization until the end of the {{init}} call, 
instead of inside the {{supplier.get}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1559: KAFKA-3844; Sort configuration items in log

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1559


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3844) Sort configuration items in log

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3844.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1559
[https://github.com/apache/kafka/pull/1559]

> Sort configuration items in log
> ---
>
> Key: KAFKA-3844
> URL: https://issues.apache.org/jira/browse/KAFKA-3844
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Trivial
> Fix For: 0.10.1.0
>
>
> Currently, the output of 
> org.apache.kafka.common.config.AbstractConfig#logAll() is unsorted, so it's 
> not convenient to check related configurations. The configuration items in 
> log could be sorted, so that related items be adjacent. For example, all 
> "log.*" configuration items would be adjacent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3844) Sort configuration items in log

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368844#comment-15368844
 ] 

ASF GitHub Bot commented on KAFKA-3844:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1559


> Sort configuration items in log
> ---
>
> Key: KAFKA-3844
> URL: https://issues.apache.org/jira/browse/KAFKA-3844
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Trivial
> Fix For: 0.10.1.0
>
>
> Currently, the output of 
> org.apache.kafka.common.config.AbstractConfig#logAll() is unsorted, so it's 
> not convenient to check related configurations. The configuration items in 
> log could be sorted, so that related items be adjacent. For example, all 
> "log.*" configuration items would be adjacent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1592: Update design.html

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1592


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1595: MINOR: Typo fix in comments

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1595


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3849) Avoid polling every second in MirrorMaker if no data is available.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368875#comment-15368875
 ] 

ASF GitHub Bot commented on KAFKA-3849:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1515


> Avoid polling every second in MirrorMaker if no data is available.
> --
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> MirrorMaker has consumer poll time, consumer timeout time for new consumer, 
> hard coded at 1000 ms. This should be configurable as it is in case of old 
> consumer. Default can stay as 1000 ms though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1515: KAFKA-3849: Add explanation on why polling every s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1515


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3849) Avoid polling every second in MirrorMaker if no data is available.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3849:
---
   Resolution: Fixed
Fix Version/s: (was: 0.10.0.1)
   0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1515
[https://github.com/apache/kafka/pull/1515]

> Avoid polling every second in MirrorMaker if no data is available.
> --
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> MirrorMaker has consumer poll time, consumer timeout time for new consumer, 
> hard coded at 1000 ms. This should be configurable as it is in case of old 
> consumer. Default can stay as 1000 ms though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3849) Add explanation on why polling every second in MirrorMaker is required

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3849:
---
Summary: Add explanation on why polling every second in MirrorMaker is 
required  (was: Avoid polling every second in MirrorMaker if no data is 
available.)

> Add explanation on why polling every second in MirrorMaker is required
> --
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> MirrorMaker has consumer poll time, consumer timeout time for new consumer, 
> hard coded at 1000 ms. This should be configurable as it is in case of old 
> consumer. Default can stay as 1000 ms though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3849) Add explanation on why polling every second in MirrorMaker is required

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3849:
---
Description: (was: MirrorMaker has consumer poll time, consumer timeout 
time for new consumer, hard coded at 1000 ms. This should be configurable as it 
is in case of old consumer. Default can stay as 1000 ms though.)

> Add explanation on why polling every second in MirrorMaker is required
> --
>
> Key: KAFKA-3849
> URL: https://issues.apache.org/jira/browse/KAFKA-3849
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient

[Dana Powers]

Thanks for the confirmation, Ismael. I will write something up for further
discussion.

-Dana
On Jul 5, 2016 4:43 PM, "Ismael Juma"  wrote:

> Hi Dana,
>
> Thanks for the PR. Technically, a (simple) KIP is required when proposing
> new configs.
>
> Ismael
>
> On Sun, Jun 19, 2016 at 7:42 PM, Dana Powers 
> wrote:
>
> > I took a stab at implementing a simplified exponential + randomized
> > backoff policy here: https://github.com/apache/kafka/pull/1523
> >
> > Rather than changing public interfaces / using plugins, it just adds a
> > new client configuration "reconnect.backoff.max" that can be used to
> > enable increasing backoff when node failures repeat. If this
> > configuration is not set higher than reconnect.backoff.ms then the
> > current constant backoff policy is retained. The default is to
> > continue w/ current 50ms constant backoff.
> >
> > Thoughts? Would a change like this require a KIP?
> >
> > -Dana
> >
> >
> > On Mon, May 2, 2016 at 10:29 AM, Guozhang Wang 
> wrote:
> > > For the specific problem of connection storm, randomized with normal
> > > distribution at specified mean as "reconnect.backoff.ms" has been
> proved
> > > pretty well. The most recent usage of it in my mind is RAFT, and it
> turns
> > > out pretty effective in eliminating leader-election storms.
> > >
> > >
> > > Guozhang
> > >
> > > On Fri, Apr 29, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > e...@confluent.io>
> > > wrote:
> > >
> > >> I'll agree w/ Jay and point out that the implementations of
> > >> ReconnectionPolicy provided built-in with that driver are Constant,
> > >> Exponential, and Counting. Constant and exponential can be combined
> with
> > >> the right set of policy config parameters. I'm curious if there's a
> real
> > >> need for something else, or if you're just looking for something
> > >> exponential instead of non-constant? I think a fixed exponential
> backoff
> > >> policy that defaults parameters to the current constant backoff policy
> > >> would probably satisfy our needs.
> > >>
> > >> On Mon, Apr 4, 2016 at 1:25 PM, Jay Kreps  wrote:
> > >>
> > >> > If I understand the problem we are fixing is a connection storm
> where
> > >> when
> > >> > a new broker comes online it is overwhelmed with connections.
> > >> >
> > >> > In general we try hard to avoid plugins where possible. Maybe
> instead
> > of
> > >> > adding another plugin interface we could just directly solve this
> > problem
> > >> > by doing some randomization in the backoff to space out the
> > >> reconnections?
> > >> > This seems like it would be good for anyone with a large client
> > >> > environment?
> > >> >
> > >> > -Jay
> > >> >
> > >> > On Mon, Apr 4, 2016 at 12:54 PM, Florian Hussonnois <
> > >> fhussonn...@gmail.com
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Kafka Team,
> > >> > >
> > >> > > I have made a new Kafka Improvement Proposal.
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient
> > >> > >
> > >> > > This is my first proposal so I don't know if I have given enough
> > >> > > information.
> > >> > > Also I have already proposed an implementation :
> > >> > > https://github.com/apache/kafka/pull/1179
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > --
> > >> > > Florian HUSSONNOIS
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Ewen
> > >>
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>

Jenkins build is back to normal : kafka-trunk-jdk8 #742

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #1412

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3849; Add explanation on why polling every second in MirrorMaker

--
[...truncated 3325 lines...]

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.in

Build failed in Jenkins: kafka-trunk-jdk8 #743

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3849; Add explanation on why polling every second in MirrorMaker

--
[...truncated 6745 lines...]

kafka.coordinator.GroupMetadataManagerTest > testStoreNonEmptyGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols STARTED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols PASSED

kafka.coordinator.MemberMetadataTest > testMetadata STARTED

kafka.coordinator.MemberMetadataTest > testMetadata PASSED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
STARTED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
PASSED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol STARTED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol PASSED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
STARTED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
PASSED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure PASSED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocol STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocol PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
PASSED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync STARTED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync PASSED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers PASSED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToDeadTransition 
STARTED

kafka.coor