Re: Auto create "chroot" on server start up?

[David Arthur]

I don't think its too much to ask people to create their ZK paths: 
zkCli.sh -server localhost:2181 create /kafka kafka


Asking a ZK application to use a non-existent chroot is like asking tar 
to extract to a non-existent directory. An error is appropriate. I think 
the fix here is to give a better error than the very unhelpful/cryptic 
"Path length must be > 0" (https://issues.apache.org/jira/browse/KAFKA-294)


Cheers
-David

On 11/13/13 11:13 PM, marek wrote:

Thank you Joel, got it.

I cloned the git repo and am working in trunk.

This is the correct branch for the latest in 0.8, is that right?

Or should I work against one of the other branches?

-Marek

On Wednesday, November 13, 2013, Joel Koshy wrote:


Marek was referring to the zookeeper namespace - Marek, there are
jira(s)
https://issues.apache.org/jira/browse/KAFKA-294
https://issues.apache.org/jira/browse/KAFKA-404

Should be straightforward to finish that up - want to give that a go?

Thanks,

Joel

On Thu, Nov 14, 2013 at 03:48:01AM +0400, Kostya Golikov wrote:

Just one word: Windows.

I'm not a maintainer, but AFAIK kafka targets Windows as well and isn't
chroot is a Unix-specific feature?


2013/11/14 marek >


Hey,

I'm trying to feel-out if auto creating the broker's registration path
on server start up would be wanted as a code contribution.

How do the maintainers feel about:

1.
Having configurable server parameter:

 auto.create.chroot = true | false

2.
If (1) is set to true, the chroot would be create automatically in the
case that the it is missing from zookeeper, and the server would start
up normal.

If the maintainers are OK with this idea I'll code it up, it would
solve a use case we have at my work, and I would send in a patch or
follow any contribution process given.


Right now if you specify zookeeper.connect as:

 host1:port1,host2:port2,host3:port3/chroot/path

and /chroot/path does not exist in zookeeper, the server does not

operate.


-Marek

Errors in the Kafka by-laws

[Noah Slater]

Hello,

Your bylaws have an erroneous definition of lazy consensus:

> Lazy consensus requires 3 binding +1 votes and no binding vetoes.

Lazy consensus is "silence is consent", and requires no +1 votes. The
correct term for what your by-laws describe here is consensus
approval.

The Apache Hadoop bylaws have the correct definitions:

http://hadoop.apache.org/bylaws.html

Hope this helps!

Thanks,

-- 
Noah Slater
https://twitter.com/nslater

[jira] [Commented] (KAFKA-1124) Sending to a new topic (with auto.create.topics.enable) returns ERROR

[Jason Rosenberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822488#comment-13822488
 ] 

Jason Rosenberg commented on KAFKA-1124:


[~anujmehta] are you looking in the logs for the producer, or the server?  The 
ERROR I see is in the producer.  It does go on to complain aobut 
LeaderNotAvailable, etc., but it also gives me the ERROR message.

I'm using 0.8, sha: cd3b79699341afb8d52c51d9ac7317d93c32eeb6  (dated Oct 16).  
Which version are you using?

Jason

> Sending to a new topic (with auto.create.topics.enable) returns ERROR
> -
>
> Key: KAFKA-1124
> URL: https://issues.apache.org/jira/browse/KAFKA-1124
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>
> I had thought this was reported issue, but can't seem to find a previous 
> report for it.
> If auto.create.topics.enable is true, a producer still gets an ERROR logged 
> on the first attempt to send a message to a new topic, e.g.:
> 2013-11-06 03:00:08,638 ERROR [Thread-1] async.DefaultEventHandler - Failed 
> to collate messages by topic, partition due to: Failed to fetch topic 
> metadata for topic: mynewtopic
> 2013-11-06 03:00:08,638  INFO [Thread-1] async.DefaultEventHandler - Back off 
> for 100 ms before retrying send. Remaining retries = 3
> This usually clears itself up immediately on retry (after 100 ms), as handled 
> by the the kafka.producer.async.DefaultEventHandler (with retries enabled).
> However, this is logged to the client as an ERROR, and looks scary, when in 
> fact it should have been a normal operation (since we have 
> auto.create.topics.enable=true).
> There should be a better interaction here between the producer client and the 
> server.
> Perhaps the server can create the topic in flight before returning the 
> metadata request.
> Or, if it needs to be asynchronous, it could return a code which indicates 
> something like: "The topic doesn't exist yet, it is being created, try again 
> shortly".and have the client automatically retry (even if retries not 
> enabled, since it's not an ERROR condition, really).
> The ERROR log level is a problem since apps often have alert systems set up 
> to notify when any ERROR happens, etc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15201: address more review comments

[Jun Rao]

> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 70
> > 
> >
> > fetchsize -> fetch-size for consistency with other options?

changed.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 74
> > 
> >
> > We can probably change this to ConsumerConfig.FetchSize. Anytime we 
> > change the max message size on the broker, we will probably change default 
> > fetch size on consumer, so that can serve as the source of truth for this 
> > tool as well.

changed.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, lines 110-111
> > 
> >
> > Should we just do .replaceAll("""["']""", "") instead?

Not long used.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, lines 165-181
> > 
> >
> > val fetcherThreads = for ((i, element) <- 
> > topicAndPartitionsPerBroker.view.zipWithIndex) yield new 
> > ReplicaFetcher(, doVerification = if (i == 0) true else false) 
> > 
> > to avoid variable = true and then variable = false?

Changed, in a different way.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, lines 242-244
> > 
> >
> > Minor comment: Can we rename this to initialOffsetMap (we use the 
> > offsetMap name in ReplicaFetchThread), I got confused on the first glace..

done.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 272
> > 
> >
> > I thought this loop is supposed to go through all the messages that can 
> > be returned by the messageIterator, but looks like it will check for only 
> > the first message obtained via each messageIterator.

This loop is just iterating one message from every replica. The outer loop 
iterates through all messages.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, lines 291-295
> > 
> >
> > Wondering why we don't exit when checksums don't match?

It's probably better to report all occurrences of mismatched checksums.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 376
> > 
> >
> > typo

changed.


> On Nov. 13, 2013, 7:43 a.m., Swapnil Ghike wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 385
> > 
> >
> > typo

changed.


On Nov. 13, 2013, 7:43 a.m., Jun Rao wrote:
> > Another caveat seems to be that the tool cannot handle changes in 1. 
> > partition leadership change 2. topic configuration change (number of 
> > partitions).

The tool doesn't care about leader changes since it has to read from every 
replica. It won't pick up newly created partitions or partitions moved by the 
reassignment tool. For those cases, we can just restart the tool.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28781
---


On Nov. 12, 2013, 4:34 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 12, 2013, 4:34 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117; fix 3
> 
> 
> kafka-1117; fix 2
> 
> 
> kafka-1117; fix 1
> 
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   config/tools-log4j.properties 79240490149835656e2a013a9702c5aa41c104f1 
>   core/src/main/scala/kafka/api/OffsetResponse.scala 
> 08dc3cd3d166efba6b2b43f6e148f636b175affe 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 15201: address more review comments

[Jun Rao]

> On Nov. 13, 2013, 2:09 a.m., Joel Koshy wrote:
> > Looks good overall - this is definitely a useful tool to have. I'm posting 
> > comments so far but I will come back to it tomorrow to finish up.
> > 
> > So what would be the recommended procedure when the tool reports an 
> > inconsistency? Would operations have to go in, stop the brokers and 
> > truncate the log segments to the last segments that are verified to be 
> > consistent? If so, it would be convenient if the tool could issue another 
> > OffsetRequest to report the offset boundaries of the last consistent 
> > segments so you don't have to work too hard to determine the segment file.
> > 
> > Also, unless I'm reading it incorrect additional caveats are that the tool 
> > can only keep working so long as: partitions are not being added (or 
> > reassigned); logs are not being compacted.
> >

The tool is meant for detecting inconsistent replicas during testing so that we 
can fix the bugs. Fixing inconsistent replicas in operation will be hard and 
will require manual work.

The limitation of the tools is that it won't pick up newly created partitions 
or partitions moved by the reassignment tool. For those cases, we can just 
restart the tool.


> On Nov. 13, 2013, 2:09 a.m., Joel Koshy wrote:
> > config/tools-log4j.properties, line 16
> > 
> >
> > Is this needed?

removed.


> On Nov. 13, 2013, 2:09 a.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 98
> > 
> >
> > This is becoming my most frequent review comment :)
> > Can you use CommandLineUtils.checkRequiredArgs?

done.


> On Nov. 13, 2013, 2:09 a.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 106
> > 
> >
> > Any reason to not use kafka.consumer.Whitelist and the isTopicAllowed 
> > API that it provides?

Yes, it can be reused.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28744
---


On Nov. 12, 2013, 4:34 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 12, 2013, 4:34 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117; fix 3
> 
> 
> kafka-1117; fix 2
> 
> 
> kafka-1117; fix 1
> 
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   config/tools-log4j.properties 79240490149835656e2a013a9702c5aa41c104f1 
>   core/src/main/scala/kafka/api/OffsetResponse.scala 
> 08dc3cd3d166efba6b2b43f6e148f636b175affe 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 15201: address all review comments

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/
---

(Updated Nov. 14, 2013, 4:24 p.m.)


Review request for kafka.


Summary (updated)
-

address all review comments


Bugs: KAFKA-1117
https://issues.apache.org/jira/browse/KAFKA-1117


Repository: kafka


Description (updated)
---

kafka-1117; fix 4


kafka-1117; fix 3


kafka-1117; fix 2


kafka-1117; fix 1


kafka-1117


Diffs (updated)
-

  core/src/main/scala/kafka/api/OffsetResponse.scala 
08dc3cd3d166efba6b2b43f6e148f636b175affe 
  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 

Diff: https://reviews.apache.org/r/15201/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Commented] (KAFKA-1117) tool for checking the consistency among replicas

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822568#comment-13822568
 ] 

Jun Rao commented on KAFKA-1117:


Updated reviewboard https://reviews.apache.org/r/15201/
 against branch origin/trunk

> tool for checking the consistency among replicas
> 
>
> Key: KAFKA-1117
> URL: https://issues.apache.org/jira/browse/KAFKA-1117
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.1
>
> Attachments: KAFKA-1117.patch, KAFKA-1117_2013-11-11_08:44:25.patch, 
> KAFKA-1117_2013-11-12_08:34:53.patch, KAFKA-1117_2013-11-14_08:24:41.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1117) tool for checking the consistency among replicas

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1117:
---

Attachment: KAFKA-1117_2013-11-14_08:24:41.patch

> tool for checking the consistency among replicas
> 
>
> Key: KAFKA-1117
> URL: https://issues.apache.org/jira/browse/KAFKA-1117
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.1
>
> Attachments: KAFKA-1117.patch, KAFKA-1117_2013-11-11_08:44:25.patch, 
> KAFKA-1117_2013-11-12_08:34:53.patch, KAFKA-1117_2013-11-14_08:24:41.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15201: address all review comments

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28872
---

Ship it!


Ship It!

- Guozhang Wang


On Nov. 14, 2013, 4:24 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 14, 2013, 4:24 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117; fix 4
> 
> 
> kafka-1117; fix 3
> 
> 
> kafka-1117; fix 2
> 
> 
> kafka-1117; fix 1
> 
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/api/OffsetResponse.scala 
> 08dc3cd3d166efba6b2b43f6e148f636b175affe 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

Re: Review Request 15201: address all review comments

[Swapnil Ghike]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15201/#review28873
---

Ship it!


Ship It!

- Swapnil Ghike


On Nov. 14, 2013, 4:24 p.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15201/
> ---
> 
> (Updated Nov. 14, 2013, 4:24 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1117
> https://issues.apache.org/jira/browse/KAFKA-1117
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> kafka-1117; fix 4
> 
> 
> kafka-1117; fix 3
> 
> 
> kafka-1117; fix 2
> 
> 
> kafka-1117; fix 1
> 
> 
> kafka-1117
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/api/OffsetResponse.scala 
> 08dc3cd3d166efba6b2b43f6e148f636b175affe 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (75 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1131  copy some more files into the release tar and zip that are 
needed/desired
https://issues.apache.org/jira/browse/KAFKA-1131
KAFKA-1130  "log.dirs" is a confusing property name
https://issues.apache.org/jira/browse/KAFKA-1130
KAFKA-1127  kafka and zookeeper server should start in daemon mode and log to 
correct position
https://issues.apache.org/jira/browse/KAFKA-1127
KAFKA-1117  tool for checking the consistency among replicas
https://issues.apache.org/jira/browse/KAFKA-1117
KAFKA-1116  Need to upgrade sbt-assembly to compile on scala 2.10.2
https://issues.apache.org/jira/browse/KAFKA-1116
KAFKA-1112  broker can not start itself after kafka is killed with -9
https://issues.apache.org/jira/browse/KAFKA-1112
KAFKA-1110  Unable to produce messages with snappy/gzip compression
https://issues.apache.org/jira/browse/KAFKA-1110
KAFKA-1109  Need to fix GC log configuration code, not able to override 
KAFKA_GC_LOG_OPTS
https://issues.apache.org/jira/browse/KAFKA-1109
KAFKA-1106  HighwaterMarkCheckpoint failure puting broker into a bad state
https://issues.apache.org/jira/browse/KAFKA-1106
KAFKA-1093  Log.getOffsetsBefore(t, …) does not return the last confirmed 
offset before t
https://issues.apache.org/jira/browse/KAFKA-1093
KAFKA-1086  Improve GetOffsetShell to find metadata automatically
https://issues.apache.org/jira/browse/KAFKA-1086
KAFKA-1082  zkclient dies after UnknownHostException in zk reconnect
https://issues.apache.org/jira/browse/KAFKA-1082
KAFKA-1079  Liars in PrimitiveApiTest that promise to test api in compression 
mode, but don't do this actually
https://issues.apache.org/jira/browse/KAFKA-1079
KAFKA-1049  Encoder implementations are required to provide an undocumented 
constructor.
https://issues.apache.org/jira/browse/KAFKA-1049
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1020  Remove getAllReplicasOnBroker from KafkaController
https://issues.apache.org/jira/browse/KAFKA-1020
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and othe

[jira] [Updated] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1112:
-

Attachment: KAFKA-1112-v2.patch

Added a new patch that addresses issues raised.

Jun
1. I don't think this is true. The check is for exact match.
2. Removed.
3. Done

Neha
1. I think I am handling this--in the case of zero we don't do a full scan.
2. Ack, fixed.
3. Well the sanity check is POST recovery. So if we have a corrupt index after 
recovery we have a bug, I don't think we should automatically try to recovery 
from this (that would be another recovery).
4. done
5. Yeah, haven't had time yet.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112-v2.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15503: Patch for KAFKA-1131

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15503/#review28903
---

Ship it!


Ship It!

- Neha Narkhede


On Nov. 14, 2013, 12:49 a.m., Joe Stein wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15503/
> ---
> 
> (Updated Nov. 14, 2013, 12:49 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1131
> https://issues.apache.org/jira/browse/KAFKA-1131
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1131 the LICENSE and NOTICE file should be in the binary release
> 
> 
> Diffs
> -
> 
>   project/Build.scala 40e0c4f3eb55a237972c6f8cfe01709453085800 
> 
> Diff: https://reviews.apache.org/r/15503/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Joe Stein
> 
>

Some Kafka conversation going on at HN right now...

[David Arthur]

https://news.ycombinator.com/item?id=6734381

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13823291#comment-13823291
 ] 

Jun Rao commented on KAFKA-1112:


The following is my confusion.

The patch relies on a potentially corrupted index to find the right starting 
position in the segment file. What if the starting position given by the last 
index entry is corrupted? Then, the position could point to the middle of a 
message in the segment file. Then, the offset value we read from the segment 
file could be anything. If that value happens to match recoverPoint - 1, we 
could think no recovery is needed, but the segment file is actually corrupted. 

Similarly, even if the index file is not corrupted, the segment file could 
still be corrupted before recoverPoint - 1 (since the unit of flushing is a 
page). It's also possible that we read a corrupted piece of data as the offset 
that happens to match recoverPoint - 1, and therefore incorrectly think that 
recovery is not needed.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112-v2.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)