[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821554#comment-13821554
 ] 

Jun Rao commented on KAFKA-1112:


Thanks for the patch. A few comments.

1. I am a bit concerned of depending on a potentially corrupted index to look 
for recoveryPoint - 1 in LogSegment.needsRecovery(). If the index points to an 
arbitrary position in FileMessageSet, the offset value that 
FileMessageSet.searchFor() finds  is garbage. If that value happens to be 
larger than targetOffset, we will assume that we find targetOffset, but in fact 
we haven't.

2. LogTest.testCorruptLog(): Is the println statement needed?

3. Could you rebase?

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821591#comment-13821591
 ] 

Neha Narkhede commented on KAFKA-1112:
--

Thanks for the patch. Few comments -

1. In the most common case of needsRecovery, the position of the last entry 
will be zero. In this case, we will search the entire log segment up until the 
recovery point. This will slow down server startup but probably only when we 
really need recovery.
2. LogSegment: We have to be carefully => We have to be careful
3. Log: If sanityCheck throws an exception, can we automatically invoke index 
rebuild instead of bailing out?
4. Could you rebase?
5. Could you give the patch review tool a spin? The setup is minimal and we can 
save time for this and future reviews - 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+patch+review+tool#Kafkapatchreviewtool-1.Setup
Usage: 
python kafka-patch-review.py -j KAFKA-1112 -b trunk

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821594#comment-13821594
 ] 

Jay Kreps commented on KAFKA-1112:
--

Actually I am attempting to cover every possible case here, so the only case 
that should go through is the one where the offset of the final message is 
recoveryPoint-1 exactly. Notice that the unit test actually runs through 50 
cases of random garbage appended to the index so assuming that test is write I 
think this does work.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821638#comment-13821638
 ] 

Guozhang Wang commented on KAFKA-1112:
--

Regarding Jun's comment #1, I am more concerned about using searchFor function 
on a FileMessageSet that might be corrupted. From the code it seems if 
FileMessageSet is corrupted the searchFor function may actually not return due 
to variable position not monotonically increasing?

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 3

[Joe Stein]

Thanks David, I am going to have to throw down a -1 on the binary
artifacts, your right we need LICENSE and NOTICE in the binary distro also
the JARs we are putting up into Maven (they are supposed to be in the
MANIFEST and their not).  I created a ticket for the former but haven't yet
for the latter ... haven't figured out how to-do that with SBT yet.

So, before I propose a new VOTE I think fixing the LICENSE and NOTICE for
the binary distro is going to take 13 seconds to fix.  Getting the LICENSE
and NOTICE into the jars that are published is going to take a little
longer.  I will start a discussion on the mailing list.

If anyone else reads our release policy different let me know
http://www.apache.org/dev/release.html

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Nov 11, 2013 at 10:06 AM, David Arthur  wrote:

> +0 overall. The quality of the build seems good, but I have a few minor
> nits/questions.
>
> 1) java.lang.NoClassDefFoundError in ./bin/kafka-simple-consumer-
> perf-test.sh
>
> $ ./bin/kafka-simple-consumer-perf-test.sh
> Exception in thread "main" java.lang.NoClassDefFoundError: kafka/perf/
> SimpleConsumerPerformance
> Caused by: java.lang.ClassNotFoundException: kafka.perf.
> SimpleConsumerPerformance
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>
> looks like the perf classes are not included. Maybe this script should be
> removed? Or we should include the perf sub-project.
>
> 2) No README or INSTALL in the binary dist
>
> 3) Missing LICENSE and NOTICE at the top level, aren't these required for
> an Apache release?
>
> 4) Artifact name - not sure on Apache's policy, but should it be
> "apache-kafka"?
>
> Regarding the actual build, I started up with the bin script, produced a
> few messages, consumed a few, and then ran the integration tests from
> kafka-python against it. Everything seems good here.
>
> -David
>
>
> On 11/8/13 3:56 PM, Joe Stein wrote:
>
>> This is the third candidate for release of Apache Kafka 0.8.0.   This
>> release candidate artifacts changed in regards to
>> https://issues.apache.org/jira/browse/KAFKA-1126 along with updating the
>> release notes to only have resolved - fixed issues and the KEY file to
>> verify signature again for this vote to be the latest link.
>>
>> Release Notes for the 0.8.0 release
>> http://people.apache.org/~joestein/kafka-0.8.0-
>> candidate3/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Wednesday November, 13th, 12pm PDT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://svn.apache.org/repos/asf/kafka/KEYS in addition to the md5 and
>> sha1
>> checksum
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://people.apache.org/~joestein/kafka-0.8.0-candidate3/
>>
>> * Maven artifacts to be voted upon prior to release:
>> https://repository.apache.org/content/groups/staging/
>>
>> (i.e. in sbt land this can be added to the build.sbt to use Kafka
>> resolvers += "Apache Staging" at "
>> https://repository.apache.org/content/groups/staging/";
>> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
>> )
>>
>> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> bc24fb7c0f48c36bf9073d384eb87213644648d4
>>
>> /***
>>   Joe Stein
>>   Founder, Principal Consultant
>>   Big Data Open Source Security LLC
>>   http://www.stealth.ly
>>   Twitter: @allthingshadoop 
>> /
>>
>>
>

[jira] [Created] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven

[Joe Stein (JIRA)]

Joe Stein created KAFKA-1133:


 Summary: LICENSE and NOTICE files need to get into  META-INF when 
jars are built before they're signed for publishing to maven
 Key: KAFKA-1133
 URL: https://issues.apache.org/jira/browse/KAFKA-1133
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8.1






--
This message was sent by Atlassian JIRA
(v6.1#6144)

[DISCUSSION] 0.8.0 release

[Joe Stein]

What do folks think about

A source release + binary server only?  I think I can do this today
https://issues.apache.org/jira/browse/KAFKA-1131 and call a new VOTE like
tomorrow?

Or wait ~ a week and do a source release + binary server + maven push?
https://issues.apache.org/jira/browse/KAFKA-1133 I can research it some
later tonight but no clue off the top of my head how to push more different
files into META-INF as we need to-do
http://www.apache.org/dev/release.htmlif someone already knows please
throw a patch up on it and I will take a
look and commit it looks good... I probably won't get to it until the
weekend but feasible for Monday... so we can try a VOTE next Monday and
wait and have things up to Maven Central for 0.8.0?



/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

[jira] [Updated] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1133:
-

Description: This needs to happen in our Build.scala the sbt package docs 
http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight 
forward line of code or ten or whatever to-do this maybe

> LICENSE and NOTICE files need to get into  META-INF when jars are built 
> before they're signed for publishing to maven
> -
>
> Key: KAFKA-1133
> URL: https://issues.apache.org/jira/browse/KAFKA-1133
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8.1
>
>
> This needs to happen in our Build.scala the sbt package docs 
> http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight 
> forward line of code or ten or whatever to-do this maybe



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [DISCUSSION] 0.8.0 release

[Joel Koshy]

Based on
http://www.apache.org/dev/release.html#distribute-other-artifacts it
seems we should close KAFKA-1133 before releasing.


"For example, the Java artifact format is based on a compressed
directory structure and those projects wishing to distribute jars must
place LICENSE and NOTICE files in the META-INF directory within the
jar"


Thanks,

Joel

On Wed, Nov 13, 2013 at 02:59:01PM -0500, Joe Stein wrote:
> What do folks think about
> 
> A source release + binary server only?  I think I can do this today
> https://issues.apache.org/jira/browse/KAFKA-1131 and call a new VOTE like
> tomorrow?
> 
> Or wait ~ a week and do a source release + binary server + maven push?
> https://issues.apache.org/jira/browse/KAFKA-1133 I can research it some
> later tonight but no clue off the top of my head how to push more different
> files into META-INF as we need to-do
> http://www.apache.org/dev/release.htmlif someone already knows please
> throw a patch up on it and I will take a
> look and commit it looks good... I probably won't get to it until the
> weekend but feasible for Monday... so we can try a VOTE next Monday and
> wait and have things up to Maven Central for 0.8.0?
> 
> 
> 
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /

Auto create "chroot" on server start up?

[marek]

Hey,

I'm trying to feel-out if auto creating the broker's registration path
on server start up would be wanted as a code contribution.

How do the maintainers feel about:

1.
Having configurable server parameter:

auto.create.chroot = true | false

2.
If (1) is set to true, the chroot would be create automatically in the
case that the it is missing from zookeeper, and the server would start
up normal.

If the maintainers are OK with this idea I'll code it up, it would
solve a use case we have at my work, and I would send in a patch or
follow any contribution process given.


Right now if you specify zookeeper.connect as:

host1:port1,host2:port2,host3:port3/chroot/path

and /chroot/path does not exist in zookeeper, the server does not operate.


-Marek

Re: Auto create "chroot" on server start up?

[Kostya Golikov]

Just one word: Windows.

I'm not a maintainer, but AFAIK kafka targets Windows as well and isn't
chroot is a Unix-specific feature?


2013/11/14 marek 

> Hey,
>
> I'm trying to feel-out if auto creating the broker's registration path
> on server start up would be wanted as a code contribution.
>
> How do the maintainers feel about:
>
> 1.
> Having configurable server parameter:
>
> auto.create.chroot = true | false
>
> 2.
> If (1) is set to true, the chroot would be create automatically in the
> case that the it is missing from zookeeper, and the server would start
> up normal.
>
> If the maintainers are OK with this idea I'll code it up, it would
> solve a use case we have at my work, and I would send in a patch or
> follow any contribution process given.
>
>
> Right now if you specify zookeeper.connect as:
>
> host1:port1,host2:port2,host3:port3/chroot/path
>
> and /chroot/path does not exist in zookeeper, the server does not operate.
>
>
> -Marek
>

Re: Auto create "chroot" on server start up?

[Joel Koshy]

Marek was referring to the zookeeper namespace - Marek, there are
jira(s)
https://issues.apache.org/jira/browse/KAFKA-294
https://issues.apache.org/jira/browse/KAFKA-404

Should be straightforward to finish that up - want to give that a go?

Thanks,

Joel

On Thu, Nov 14, 2013 at 03:48:01AM +0400, Kostya Golikov wrote:
> Just one word: Windows.
> 
> I'm not a maintainer, but AFAIK kafka targets Windows as well and isn't
> chroot is a Unix-specific feature?
> 
> 
> 2013/11/14 marek 
> 
> > Hey,
> >
> > I'm trying to feel-out if auto creating the broker's registration path
> > on server start up would be wanted as a code contribution.
> >
> > How do the maintainers feel about:
> >
> > 1.
> > Having configurable server parameter:
> >
> > auto.create.chroot = true | false
> >
> > 2.
> > If (1) is set to true, the chroot would be create automatically in the
> > case that the it is missing from zookeeper, and the server would start
> > up normal.
> >
> > If the maintainers are OK with this idea I'll code it up, it would
> > solve a use case we have at my work, and I would send in a patch or
> > follow any contribution process given.
> >
> >
> > Right now if you specify zookeeper.connect as:
> >
> > host1:port1,host2:port2,host3:port3/chroot/path
> >
> > and /chroot/path does not exist in zookeeper, the server does not operate.
> >
> >
> > -Marek
> >

Re: Auto create "chroot" on server start up?

[marek]

Hello Kostya,

The "chroot" that the zookeeper.connect parameter takes as a suffix is just
an analogy to the UNIX command "chroot", but it has nothing to do with
actually using the command line tool.

For clarity check out the "broker config" section at:

http://kafka.apache.org/documentation.html


Hopefully it will make sense.


On Wednesday, November 13, 2013, Kostya Golikov wrote:

> Just one word: Windows.
>
> I'm not a maintainer, but AFAIK kafka targets Windows as well and isn't
> chroot is a Unix-specific feature?
>
>
> 2013/11/14 marek >
>
> > Hey,
> >
> > I'm trying to feel-out if auto creating the broker's registration path
> > on server start up would be wanted as a code contribution.
> >
> > How do the maintainers feel about:
> >
> > 1.
> > Having configurable server parameter:
> >
> > auto.create.chroot = true | false
> >
> > 2.
> > If (1) is set to true, the chroot would be create automatically in the
> > case that the it is missing from zookeeper, and the server would start
> > up normal.
> >
> > If the maintainers are OK with this idea I'll code it up, it would
> > solve a use case we have at my work, and I would send in a patch or
> > follow any contribution process given.
> >
> >
> > Right now if you specify zookeeper.connect as:
> >
> > host1:port1,host2:port2,host3:port3/chroot/path
> >
> > and /chroot/path does not exist in zookeeper, the server does not
> operate.
> >
> >
> > -Marek
> >
>

Review Request 15503: Patch for KAFKA-1131

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15503/
---

Review request for kafka.


Bugs: KAFKA-1131
https://issues.apache.org/jira/browse/KAFKA-1131


Repository: kafka


Description
---

KAFKA-1131 the LICENSE and NOTICE file should be in the binary release


Diffs
-

  project/Build.scala 40e0c4f3eb55a237972c6f8cfe01709453085800 

Diff: https://reviews.apache.org/r/15503/diff/


Testing
---


Thanks,

Joe Stein

[jira] [Commented] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822039#comment-13822039
 ] 

Joe Stein commented on KAFKA-1131:
--

Created reviewboard https://reviews.apache.org/r/15503/
 against branch trunk

> copy some more files into the release tar and zip that are needed/desired
> -
>
> Key: KAFKA-1131
> URL: https://issues.apache.org/jira/browse/KAFKA-1131
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
> Fix For: 0.8.1
>
> Attachments: KAFKA-1131.patch
>
>
> e.g.  LICENSE, NOTICE, README (or maybe a new quick start file .txt) 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1131:
-

Attachment: KAFKA-1131.patch

> copy some more files into the release tar and zip that are needed/desired
> -
>
> Key: KAFKA-1131
> URL: https://issues.apache.org/jira/browse/KAFKA-1131
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
> Fix For: 0.8.1
>
> Attachments: KAFKA-1131.patch
>
>
> e.g.  LICENSE, NOTICE, README (or maybe a new quick start file .txt) 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1131:
-

Fix Version/s: 0.8

> copy some more files into the release tar and zip that are needed/desired
> -
>
> Key: KAFKA-1131
> URL: https://issues.apache.org/jira/browse/KAFKA-1131
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
> Fix For: 0.8, 0.8.1
>
> Attachments: KAFKA-1131.patch
>
>
> e.g.  LICENSE, NOTICE, README (or maybe a new quick start file .txt) 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15503: Patch for KAFKA-1131

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15503/#review28851
---

Ship it!


Ship It!

- Guozhang Wang


On Nov. 14, 2013, 12:49 a.m., Joe Stein wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15503/
> ---
> 
> (Updated Nov. 14, 2013, 12:49 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1131
> https://issues.apache.org/jira/browse/KAFKA-1131
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1131 the LICENSE and NOTICE file should be in the binary release
> 
> 
> Diffs
> -
> 
>   project/Build.scala 40e0c4f3eb55a237972c6f8cfe01709453085800 
> 
> Diff: https://reviews.apache.org/r/15503/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Joe Stein
> 
>

[jira] [Commented] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired

[David Arthur (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822072#comment-13822072
 ] 

David Arthur commented on KAFKA-1131:
-

[~charmalloc], patch looks good, verified that it adds NOTICE and LICENSE in 
the release directory.

Two quick questions:

* Do we need to include these in the jar for the binary dist, or just for Maven?
* If I'm reading http://apache.org/legal/src-headers.html#notice right, we need 
to include 3rd party licenses in the NOTICE file of the binary dist (since we 
are distributing those 3rd party libs). I know Ivy can generate a list of all 
the licenses, so SBT probably can as well.

> copy some more files into the release tar and zip that are needed/desired
> -
>
> Key: KAFKA-1131
> URL: https://issues.apache.org/jira/browse/KAFKA-1131
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
> Fix For: 0.8, 0.8.1
>
> Attachments: KAFKA-1131.patch
>
>
> e.g.  LICENSE, NOTICE, README (or maybe a new quick start file .txt) 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [DISCUSSION] 0.8.0 release

[David Arthur]

BTW, we should probably keep a set of release notes on the wiki so this 
process can get easier in the future, and not live fully in Joe's head ;)


-David

On 11/13/13 6:05 PM, Joel Koshy wrote:

Based on
http://www.apache.org/dev/release.html#distribute-other-artifacts it
seems we should close KAFKA-1133 before releasing.


"For example, the Java artifact format is based on a compressed
directory structure and those projects wishing to distribute jars must
place LICENSE and NOTICE files in the META-INF directory within the
jar"


Thanks,

Joel

On Wed, Nov 13, 2013 at 02:59:01PM -0500, Joe Stein wrote:

What do folks think about

A source release + binary server only?  I think I can do this today
https://issues.apache.org/jira/browse/KAFKA-1131 and call a new VOTE like
tomorrow?

Or wait ~ a week and do a source release + binary server + maven push?
https://issues.apache.org/jira/browse/KAFKA-1133 I can research it some
later tonight but no clue off the top of my head how to push more different
files into META-INF as we need to-do
http://www.apache.org/dev/release.htmlif someone already knows please
throw a patch up on it and I will take a
look and commit it looks good... I probably won't get to it until the
weekend but feasible for Monday... so we can try a VOTE next Monday and
wait and have things up to Maven Central for 0.8.0?



/***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop 
/

Re: [DISCUSSION] 0.8.0 release

[Joe Stein]

Will do my best to document for RC4


On Wed, Nov 13, 2013 at 8:28 PM, David Arthur  wrote:

> BTW, we should probably keep a set of release notes on the wiki so this
> process can get easier in the future, and not live fully in Joe's head ;)
>
> -David
>
>
> On 11/13/13 6:05 PM, Joel Koshy wrote:
>
>> Based on
>> http://www.apache.org/dev/release.html#distribute-other-artifacts it
>> seems we should close KAFKA-1133 before releasing.
>>
>> 
>> "For example, the Java artifact format is based on a compressed
>> directory structure and those projects wishing to distribute jars must
>> place LICENSE and NOTICE files in the META-INF directory within the
>> jar"
>> 
>>
>> Thanks,
>>
>> Joel
>>
>> On Wed, Nov 13, 2013 at 02:59:01PM -0500, Joe Stein wrote:
>>
>>> What do folks think about
>>>
>>> A source release + binary server only?  I think I can do this today
>>> https://issues.apache.org/jira/browse/KAFKA-1131 and call a new VOTE
>>> like
>>> tomorrow?
>>>
>>> Or wait ~ a week and do a source release + binary server + maven push?
>>> https://issues.apache.org/jira/browse/KAFKA-1133 I can research it some
>>> later tonight but no clue off the top of my head how to push more
>>> different
>>> files into META-INF as we need to-do
>>> http://www.apache.org/dev/release.htmlif someone already knows please
>>> throw a patch up on it and I will take a
>>> look and commit it looks good... I probably won't get to it until the
>>> weekend but feasible for Monday... so we can try a VOTE next Monday and
>>> wait and have things up to Maven Central for 0.8.0?
>>>
>>>
>>>
>>> /***
>>>   Joe Stein
>>>   Founder, Principal Consultant
>>>   Big Data Open Source Security LLC
>>>   http://www.stealth.ly
>>>   Twitter: @allthingshadoop 
>>> /
>>>
>>
>

[jira] [Commented] (KAFKA-1131) copy some more files into the release tar and zip that are needed/desired

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822088#comment-13822088
 ] 

Joe Stein commented on KAFKA-1131:
--

<< Do we need to include these in the jar for the binary dist, or just for 
Maven?
probably, yeah, good point

<< If I'm reading http://apache.org/legal/src-headers.html#notice right, we 
need to include 3rd party licenses in the NOTICE file of the binary dist (since 
we are distributing those 3rd party libs). I know Ivy can generate a list of 
all the licenses, so SBT probably can as well.

this should all be correct as we haven't made any changes since incubation and 
it was resolved then.  we are only bumping into these jar issues now because we 
only released and signed source for 0.7.X

> copy some more files into the release tar and zip that are needed/desired
> -
>
> Key: KAFKA-1131
> URL: https://issues.apache.org/jira/browse/KAFKA-1131
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
> Fix For: 0.8, 0.8.1
>
> Attachments: KAFKA-1131.patch
>
>
> e.g.  LICENSE, NOTICE, README (or maybe a new quick start file .txt) 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Assigned] (KAFKA-1103) Consumer uses two zkclients

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-1103:


Assignee: Guozhang Wang

> Consumer uses two zkclients
> ---
>
> Key: KAFKA-1103
> URL: https://issues.apache.org/jira/browse/KAFKA-1103
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
> Fix For: 0.8.1
>
>
> .. which is very confusing when debugging consumer logs. I don't remember any 
> good reason for this, and we should get rid of the one instantiated in 
> ZookeeperTopicEventWatcher if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Auto create "chroot" on server start up?

[marek]

Thank you Joel, got it.

I cloned the git repo and am working in trunk.

This is the correct branch for the latest in 0.8, is that right?

Or should I work against one of the other branches?

-Marek

On Wednesday, November 13, 2013, Joel Koshy wrote:

> Marek was referring to the zookeeper namespace - Marek, there are
> jira(s)
> https://issues.apache.org/jira/browse/KAFKA-294
> https://issues.apache.org/jira/browse/KAFKA-404
>
> Should be straightforward to finish that up - want to give that a go?
>
> Thanks,
>
> Joel
>
> On Thu, Nov 14, 2013 at 03:48:01AM +0400, Kostya Golikov wrote:
> > Just one word: Windows.
> >
> > I'm not a maintainer, but AFAIK kafka targets Windows as well and isn't
> > chroot is a Unix-specific feature?
> >
> >
> > 2013/11/14 marek >
> >
> > > Hey,
> > >
> > > I'm trying to feel-out if auto creating the broker's registration path
> > > on server start up would be wanted as a code contribution.
> > >
> > > How do the maintainers feel about:
> > >
> > > 1.
> > > Having configurable server parameter:
> > >
> > > auto.create.chroot = true | false
> > >
> > > 2.
> > > If (1) is set to true, the chroot would be create automatically in the
> > > case that the it is missing from zookeeper, and the server would start
> > > up normal.
> > >
> > > If the maintainers are OK with this idea I'll code it up, it would
> > > solve a use case we have at my work, and I would send in a patch or
> > > follow any contribution process given.
> > >
> > >
> > > Right now if you specify zookeeper.connect as:
> > >
> > > host1:port1,host2:port2,host3:port3/chroot/path
> > >
> > > and /chroot/path does not exist in zookeeper, the server does not
> operate.
> > >
> > >
> > > -Marek
> > >
>
>

[jira] [Commented] (KAFKA-1124) Sending to a new topic (with auto.create.topics.enable) returns ERROR

[Anuj Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822177#comment-13822177
 ] 

Anuj Mehta commented on KAFKA-1124:
---

Hi

I tried reproducing this issue but I am getting a WARN message instead of an 
ERROR

[2013-11-13 18:24:59,310] WARN Error while fetching metadata [{TopicMetadata 
for topic newTopic -> 
No partition metadata for topic newTopic due to 
kafka.common.LeaderNotAvailableException}] for topic [newTopic]: class 
kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)

> Sending to a new topic (with auto.create.topics.enable) returns ERROR
> -
>
> Key: KAFKA-1124
> URL: https://issues.apache.org/jira/browse/KAFKA-1124
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>
> I had thought this was reported issue, but can't seem to find a previous 
> report for it.
> If auto.create.topics.enable is true, a producer still gets an ERROR logged 
> on the first attempt to send a message to a new topic, e.g.:
> 2013-11-06 03:00:08,638 ERROR [Thread-1] async.DefaultEventHandler - Failed 
> to collate messages by topic, partition due to: Failed to fetch topic 
> metadata for topic: mynewtopic
> 2013-11-06 03:00:08,638  INFO [Thread-1] async.DefaultEventHandler - Back off 
> for 100 ms before retrying send. Remaining retries = 3
> This usually clears itself up immediately on retry (after 100 ms), as handled 
> by the the kafka.producer.async.DefaultEventHandler (with retries enabled).
> However, this is logged to the client as an ERROR, and looks scary, when in 
> fact it should have been a normal operation (since we have 
> auto.create.topics.enable=true).
> There should be a better interaction here between the producer client and the 
> server.
> Perhaps the server can create the topic in flight before returning the 
> metadata request.
> Or, if it needs to be asynchronous, it could return a code which indicates 
> something like: "The topic doesn't exist yet, it is being created, try again 
> shortly".and have the client automatically retry (even if retries not 
> enabled, since it's not an ERROR condition, really).
> The ERROR log level is a problem since apps often have alert systems set up 
> to notify when any ERROR happens, etc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[David Lao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822188#comment-13822188
 ] 

David Lao commented on KAFKA-1112:
--

I hitting this exact issue. For what it's worth the content of the corrupt 
index file is consist of 00's for the entire file.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1112) broker can not start itself after kafka is killed with -9

[David Lao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1386#comment-1386
 ] 

David Lao commented on KAFKA-1112:
--

Jay , can you provide a patch for the 0.8 branch as well? Thanks.

> broker can not start itself after kafka is killed with -9
> -
>
> Key: KAFKA-1112
> URL: https://issues.apache.org/jira/browse/KAFKA-1112
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8, 0.8.1
>Reporter: Kane Kim
>Assignee: Jay Kreps
>Priority: Critical
> Attachments: KAFKA-1112-v1.patch, KAFKA-1112.out
>
>
> When I kill kafka with -9, broker cannot start itself because of corrupted 
> index logs. I think kafka should try to delete/rebuild indexes itself without 
> manual intervention. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Liu updated KAFKA-1127:
---

Attachment: kafka-1127-v3.patch

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127-v2.patch, kafka-1127-v3.patch, 
> kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822231#comment-13822231
 ] 

Raymond Liu commented on KAFKA-1127:


v3 patch enable to choose whether to start server in daemon mode or not. by 
default, not change the original behavior ( say server run in fg )

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127-v2.patch, kafka-1127-v3.patch, 
> kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1133) LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822234#comment-13822234
 ] 

Joe Stein commented on KAFKA-1133:
--

the archive function might be what we need to hook into 
http://www.scala-sbt.org/release/sxr/sbt/IO.scala.html#sbt.IO.archive.manifest

> LICENSE and NOTICE files need to get into  META-INF when jars are built 
> before they're signed for publishing to maven
> -
>
> Key: KAFKA-1133
> URL: https://issues.apache.org/jira/browse/KAFKA-1133
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8.1
>
>
> This needs to happen in our Build.scala the sbt package docs 
> http://www.scala-sbt.org/release/docs/Howto/package.html probably a straight 
> forward line of code or ten or whatever to-do this maybe



--
This message was sent by Atlassian JIRA
(v6.1#6144)