Re: Review Request 13908: Initial patch KAFKA-1012

2013-09-05 Thread Tejas Patil 


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/common/ErrorMapping.scala, line 45
> > 
> >
> > Could this error code be renamed to something like 
> > OffsetLoadingNotCompleteCode. Arguably this will convey the error code more 
> > clearly.

Agree. Would include change in v3 patch.


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala, line 
> > 280
> > 
> >
> > It will be good to be specific about which channel the consumer failed 
> > to establish. In this case, let's mention "Unable to establish a channel 
> > for fetching offsets with any of the live brokers in 
> > %s".format(brokers.mkString(','))

Agree. Would include change in v3 patch.


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala, line 
> > 321
> > 
> >
> > Is it a good idea for commitOffsets() to eat up every error that it 
> > encounters ? commitOffsets() is a public API and users want to use it to 
> > commit offsets on demand, manually. These users do not use auto commit 
> > offsets and use commitOffsets() to checkpoint offsets as often as the 
> > application logic dictates. For that use case, if the commitOffsets() has 
> > not actually successfully committed the offsets, the user of the API must 
> > know about it and retry as required. Thoughts?

Correct me if I am wrong: the Producer API does not expose failures to outside 
world. In case of failures, producer would internally retry the failed messages 
but thats behind the hood and would not be visible to the caller. With embedded 
producer, I could not find a way so that consumers would about failures w/o 
modifying the producer code. As "embedded producer" was a temporary hack 
thingy, we refrained from doing modifications in producer code to expose this 
info. This could be something that can be handled in phase #2 ie. using 
OffsetCommitRequest.


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala, line 
> > 664
> > 
> >
> > It is probably better to be clearer on this error message as well. 
> > Something along the lines of "as offset bootstrap is still in progress on 
> > some brokers. This means leadership changed recently for the offsets topic"

This is one of the points that Guozhang raised in his review comment 27.3 and 
now you; strong indication that I have got to change that sloppy message :) The 
loading process would be triggered by (a) broker startup and (b) leadership 
change. I tried to capture both these things in a log message but it looked 
ugly as it was too big to fit in one line. Technically 'broker startup' leads 
to leadership assignment which can be also seen as leadership change. With that 
argument, if we have to not distinguish between (a) and (b), then we could go 
with the message you suggested. Else, change the last part in your suggestion 
to "This means leadership changed recently for the offsets topic or the broker 
is starting up".


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/server/KafkaServer.scala, line 88
> > 
> >
> > Curious - why do we need to use the singleton pattern here? Shouldn't 
> > only one thread invoke KafkaServer.startup?

Its made singleton so that even if someone carelessly tries to create multiple 
offset managers on same server instance, there would still be a single offset 
manager. I agree that in current code this will not happen. The penalty of 
having multiple copies offset managers is huge in terms of memory and 
correctness, so made it singleton.


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/server/OffsetManager.scala, line 17
> > 
> >
> > this file has turned into a big blob of code. It will help if you can 
> > separate the OffsetManager trait, the DefaultOffsetManager and 
> > ZookeeperOffsetManager into separate files

Agree. Would include change in v3 patch.


> On Sept. 4, 2013, 11:58 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/server/OffsetManager.scala, line 51
> > 
> >
> > I think it is best to not include any parameters to the startup() API 
> > as it is difficult to come up with a set of parameters that would work for 
> > all possible offset managers. What might work better is to include a 
> > generic init API that takes in a P

[jira] [Closed] (KAFKA-1039) kafka return acks but not logging while sending messages with compression

2013-09-05 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao closed KAFKA-1039.
--


> kafka return acks but not logging while sending messages with compression
> -
>
> Key: KAFKA-1039
> URL: https://issues.apache.org/jira/browse/KAFKA-1039
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8
> Environment: ubuntu 64bit
>Reporter: Xiang chao
>Assignee: Jay Kreps
>
> when send message with compression, the broker return acks, but don't write 
> messages to disk. So I can't get messages using consumers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (KAFKA-1039) kafka return acks but not logging while sending messages with compression

2013-09-05 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1039.


Resolution: Duplicate

Close this since it's a duplicate of kafka-1037.

> kafka return acks but not logging while sending messages with compression
> -
>
> Key: KAFKA-1039
> URL: https://issues.apache.org/jira/browse/KAFKA-1039
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8
> Environment: ubuntu 64bit
>Reporter: Xiang chao
>Assignee: Jay Kreps
>
> when send message with compression, the broker return acks, but don't write 
> messages to disk. So I can't get messages using consumers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-09-05 Thread Jun Rao (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759134#comment-13759134
 ] 

Jun Rao commented on KAFKA-1036:


Hi, Tim,

Do you still want to provide a patch? I saw a patch attached and then deleted.

Thanks,

> Unable to rename replication offset checkpoint in windows
> -
>
> Key: KAFKA-1036
> URL: https://issues.apache.org/jira/browse/KAFKA-1036
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: windows
>Reporter: Timothy Chen
>Priority: Critical
>  Labels: windows
>
> Although there was a fix for checkpoint file renaming in windows that tries 
> to delete the existing checkpoint file if renamed failed, I'm still seeing 
> renaming errors on windows even though the destination file doesn't exist.
> A bit investigation shows that it wasn't able to rename the file since the 
> kafka jvm still holds a fie lock on the tmp file and wasn't able to rename 
> it. 
> Attaching a patch that calls a explict writer.close so it can release the 
> lock and can able to rename it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-09-05 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1035:
---

Resolution: Fixed
  Assignee: Rajasekar Elango  (was: Jun Rao)
Status: Resolved  (was: Patch Available)

Thanks for the patch. +1. Committed to 0.8.

> Add message-send-max-retries and retry-backoff-ms options to console producer
> -
>
> Key: KAFKA-1035
> URL: https://issues.apache.org/jira/browse/KAFKA-1035
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.7, 0.8
>Reporter: Rajasekar Elango
>Assignee: Rajasekar Elango
>  Labels: patch
> Fix For: 0.8
>
> Attachments: console_producer_add_options.patch, 
> console_producer_add_options-v2.patch
>
>
> It's possible for console producer to give up too soon if it can't find a 
> leader of a topic. Increasing message-send-max-retries would resolve this 
> but.  Console producer doesn't provide options to set 
> message-send-max-retries and retry-backoff-ms. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-09-05 Thread Jun Rao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao closed KAFKA-1035.
--


> Add message-send-max-retries and retry-backoff-ms options to console producer
> -
>
> Key: KAFKA-1035
> URL: https://issues.apache.org/jira/browse/KAFKA-1035
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.7, 0.8
>Reporter: Rajasekar Elango
>Assignee: Rajasekar Elango
>  Labels: patch
> Fix For: 0.8
>
> Attachments: console_producer_add_options.patch, 
> console_producer_add_options-v2.patch
>
>
> It's possible for console producer to give up too soon if it can't find a 
> leader of a topic. Increasing message-send-max-retries would resolve this 
> but.  Console producer doesn't provide options to set 
> message-send-max-retries and retry-backoff-ms. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-998) Producer should not retry on non-recoverable error codes

2013-09-05 Thread Joel Koshy (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759257#comment-13759257
 ] 

Joel Koshy commented on KAFKA-998:
--

Oh I thought this was for 0.8 - it does apply on trunk.  Do people think this 
is small and important enough to apply to 0.8?


Another comment after thinking about 
the patch: in dispatchSerializedData - would it be better to just drop data 
that have hit the message size limit?  That way, there is no need to return the 
needRetry, so the dispatchSerializedData signature remains the same. The 
disadvantage is that we won't propagage a failedtosendmessage exception for 
such messages to the caller - for the producer in async mode that is probably 
fine (since right now the caller can't really do much with that exception) - in 
sync mode the caller could perhaps decide to send fewer messages at once. Even 
in that case we don't really say which topics/messages hit the message size 
limit so I think it is fine in that case as well. Furthermore, this would be 
covered by KAFKA-1026 to a large degree.

> Producer should not retry on non-recoverable error codes
> 
>
> Key: KAFKA-998
> URL: https://issues.apache.org/jira/browse/KAFKA-998
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8, 0.8.1
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Attachments: KAFKA-998.v1.patch
>
>
> Based on a discussion with Guozhang. The producer currently retries on all 
> error codes (including messagesizetoolarge which is pointless to retry on). 
> This can slow down the producer unnecessarily.
> If at all we want to retry on that error code we would need to retry with a 
> smaller batch size, but that's a separate discussion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1043) Time-consuming FetchRequest could block other request in the response queue

2013-09-05 Thread Guozhang Wang (JIRA) 
Guozhang Wang created KAFKA-1043:


 Summary: Time-consuming FetchRequest could block other request in 
the response queue
 Key: KAFKA-1043
 URL: https://issues.apache.org/jira/browse/KAFKA-1043
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.8, 0.8.1


Since in SocketServer the processor who takes any request is also responsible 
for writing the response for that request, we make each processor owning its 
own response queue. If a FetchRequest takes irregularly long time to write the 
channel buffer it would block all other responses in the queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Subscription: outstanding kafka patches

2013-09-05 Thread jira 
Issue Subscription
Filter: outstanding kafka patches (70 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1042  Fix segment flush logic
https://issues.apache.org/jira/browse/KAFKA-1042
KAFKA-1038  fetch response should use empty messageset instead of null when 
handling errors
https://issues.apache.org/jira/browse/KAFKA-1038
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1020  Remove getAllReplicasOnBroker from KafkaController
https://issues.apache.org/jira/browse/KAFKA-1020
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1008  Unmap before resizing
https://issues.apache.org/jira/browse/KAFKA-1008
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-995   Enforce that the value for replica.fetch.max.bytes is always >= the 
value for message.max.bytes
https://issues.apache.org/jira/browse/KAFKA-995
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-982   Logo for Kafka
https://issues.apache.org/jira/browse/KAFKA-982
KAFKA-981   Unable to pull Kafka binaries with Ivy
https://issues.apache.org/jira/browse/KAFKA-981
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-956   High-level consumer fails to check topic metadata response for 
errors
https://issues.apache.org/jira/browse/KAFKA-956
KAFKA-946   Kafka Hadoop Consumer fails when verifying message checksum
https://issues.apache.org/jira/browse/KAFKA-946
KAFKA-923   Improve controller failover latency
https://issues.apache.org/jira/browse/KAFKA-923
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and other kafka specific code from the 
RequestChannel
https://issues.apache.org/jira/browse/KAFKA-745
KAFKA-735   Add looping and JSON output for ConsumerOffsetChecker
https://issues.apache.org/jira/browse/KAFKA-735
KAFKA-717   scala 2.10 build support
https://issues.apache.org/jira/browse/KAFKA-717
KAFKA-686   0.8 Kafka broker should give a better error message when running 
against 0.7 zookeeper
https://issues.apache.org/jira/browse/KAFKA-686
KAFKA-674   Clean Shutdown Testing - Log segments checksums mismatch
https://issues.apache.org/jira/browse/KAFKA-674
KAFKA-652   Create testcases for clean shut-down
https://issues.apache.org/jira/browse/KAFKA-652
KAFKA-649   Cleanup log4j logging
https://issues.apache.org/jira/browse/KAFKA-649
KAFKA-645   Create a she

[jira] [Created] (KAFKA-1044) change log4j to slf4j

2013-09-05 Thread sjk (JIRA) 
sjk created KAFKA-1044:
--

 Summary: change log4j to slf4j 
 Key: KAFKA-1044
 URL: https://issues.apache.org/jira/browse/KAFKA-1044
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.8
Reporter: sjk
Assignee: Jay Kreps


can u chanage the log4j to slf4j, in my project, i use logback, it's conflict 
with log4j.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1045) producer zk.connect config

2013-09-05 Thread sjk (JIRA) 
sjk created KAFKA-1045:
--

 Summary: producer zk.connect config
 Key: KAFKA-1045
 URL: https://issues.apache.org/jira/browse/KAFKA-1045
 Project: Kafka
  Issue Type: Bug
Reporter: sjk


java.lang.IllegalArgumentException: requirement failed: Missing required 
property 'metadata.broker.list'


props.put("zk.connect", KafkaConfig.getZooAddress());

when i config zk, why the above tip appear?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

C++ client library for Apache Kafka v0.8+

2013-09-05 Thread David Tompkins 
I had a need for a C++ client API for Apache Kafka with support for the new 
v0.8 wire protocol, and I didn’t see one out in the wild yet. So I found some 
time and wrote one. You can find it here on github:

https://github.com/adobe-research/libkafka

Released under an Apache 2.0 license as open source from my current employer 
(Adobe Research). I plan to provide ongoing support for this library.,

-DT

[jira] [Commented] (KAFKA-1045) producer zk.connect config

2013-09-05 Thread Neha Narkhede (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759758#comment-13759758
 ] 

Neha Narkhede commented on KAFKA-1045:
--

Zookeeper is not required for the Kafka producer anymore, starting Kafka 08. We 
now require you to plug in a list of brokers. The configuration options are 
described here - http://kafka.apache.org/08/configuration.html

> producer zk.connect config
> --
>
> Key: KAFKA-1045
> URL: https://issues.apache.org/jira/browse/KAFKA-1045
> Project: Kafka
>  Issue Type: Bug
>Reporter: sjk
>
> java.lang.IllegalArgumentException: requirement failed: Missing required 
> property 'metadata.broker.list'
> props.put("zk.connect", KafkaConfig.getZooAddress());
> when i config zk, why the above tip appear?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: C++ client library for Apache Kafka v0.8+

2013-09-05 Thread Jay Kreps 
This is great! Thanks for the contribution!

-jay

On Thursday, September 5, 2013, David Tompkins wrote:

> I had a need for a C++ client API for Apache Kafka with support for the
> new v0.8 wire protocol, and I didn’t see one out in the wild yet. So I
> found some time and wrote one. You can find it here on github:
>
> https://github.com/adobe-research/libkafka
>
> Released under an Apache 2.0 license as open source from my current
> employer (Adobe Research). I plan to provide ongoing support for this
> library.,
>
> -DT
>
>

[jira] [Updated] (KAFKA-1006) Mirror maker loses messages of a new topic

2013-09-05 Thread Swapnil Ghike (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1006:
-

Description: 
Consumer currently uses auto.offset.reset = largest by default. If a new topic 
is created, consumer's topic watcher is fired. The consumer will first finish 
partition reassignment as part of rebalance and then start consuming from the 
tail of each partition. Until the partition reassignment is over, the server 
may have appended new messages to the new topic, consumer won't consume these 
messages. Thus, multiple batches of messages may be lost when a topic is newly 
created. 

The fix is to start consuming from the earliest offset for newly created topics.

  was:
Mirror maker currently uses auto.offset.reset = largest on the consumer side by 
default. If a new topic is created, consumer's topic watcher is fired. The 
consumer will first finish partition reassignment as part of rebalance and then 
start consuming from the tail of each partition. Until the partition 
reassignment is over, the server may have appended new messages to the new 
topic, mirror maker won't consume these messages. Thus, multiple batches of 
messages may be lost when a topic is newly created.

The fix is to start consuming from the earliest offset for newly created topics.


> Mirror maker loses messages of a new topic
> --
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>
> Consumer currently uses auto.offset.reset = largest by default. If a new 
> topic is created, consumer's topic watcher is fired. The consumer will first 
> finish partition reassignment as part of rebalance and then start consuming 
> from the tail of each partition. Until the partition reassignment is over, 
> the server may have appended new messages to the new topic, consumer won't 
> consume these messages. Thus, multiple batches of messages may be lost when a 
> topic is newly created. 
> The fix is to start consuming from the earliest offset for newly created 
> topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1006) Consumer loses messages of a new topic with auto.offset.reset = largest

2013-09-05 Thread Swapnil Ghike (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1006:
-

Summary: Consumer loses messages of a new topic with auto.offset.reset = 
largest  (was: Mirror maker loses messages of a new topic)

> Consumer loses messages of a new topic with auto.offset.reset = largest
> ---
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Swapnil Ghike
>
> Consumer currently uses auto.offset.reset = largest by default. If a new 
> topic is created, consumer's topic watcher is fired. The consumer will first 
> finish partition reassignment as part of rebalance and then start consuming 
> from the tail of each partition. Until the partition reassignment is over, 
> the server may have appended new messages to the new topic, consumer won't 
> consume these messages. Thus, multiple batches of messages may be lost when a 
> topic is newly created. 
> The fix is to start consuming from the earliest offset for newly created 
> topics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira