Re: [jira] [Commented] (KAFKA-797) Cannot start Kafka 0.8 per Quick Start instructions
Hi David, That's indeed the problem, the Scala runtime is contained in those jars currently missing in your environment. I'm not sure why they are not there automatically, it might be some setup aspect of sbt. I was under the impression sbt would make ivy fetch all necessary libraries. To your other question, *core/lib* are external dependencies. *core/target *is where sbt will place the Kafka package when building kafka via './sbt package', as well as where it will place classes if you just compile Kafka rather than packaging it into a jar. I believe *lib* and *target* serve all 'sub-projects' contained in the repo - *core*, *perf*, *examples*, and *contrib. *The later provide performance testing, java examples, and hadoop extensions, respectively. For running Kafka, core is typically enough. That's why when you run ./sbt for compiling or packaging, the default project is core. Sorry about the suggestion for compiling with 2.9.2 before, which was not compatible with the run scripts. Matan On Mon, Mar 11, 2013 at 6:07 PM, David Arthur (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598926#comment-13598926] > > David Arthur commented on KAFKA-797: > > > So upon further inspection, I see a couple of problems: > > 1. 2.8.0 is hard coded in the kafka-run-class.sh script > 2. Even when compiling with 2.8.0, the script doesn't find scala-compiler > or scala-library in the Ivy cache. It builds: > /Users/mumrah/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.8.0.jar > and > ~/Users/mumrah/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar, > but neither exist > > Building up classpaths from the Ivy cache without really using Ivy is very > brittle, IMO. > > I would be willing to work on a fix for this, but I'd need clarification > on the placement and usage of the libraries throughout the project. E.g., > what's the difference between core/lib, lib, core/target, target? > > > Cannot start Kafka 0.8 per Quick Start instructions > > --- > > > > Key: KAFKA-797 > > URL: https://issues.apache.org/jira/browse/KAFKA-797 > > Project: Kafka > > Issue Type: Bug > >Affects Versions: 0.8 > >Reporter: David Arthur > > > > Checked out latest 0.8 (b5edbb193b33ebf0b3056935a537967ff21478a6), and > ran: > > ./sbt update > > ./sbt package > > Both complete successfully. Then I try to start the server: > > $ ./bin/kafka-server-start.sh config/server.properties > > Exception in thread "main" java.lang.NoClassDefFoundError: > scala/ScalaObject > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > > at kafka.Kafka.main(Kafka.scala) > > Caused by: java.lang.ClassNotFoundException: scala.ScalaObject > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(Clas
[jira] [Commented] (KAFKA-496) high level producer send should return a response
[ https://issues.apache.org/jira/browse/KAFKA-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593775#comment-13593775 ] Matan Safriel commented on KAFKA-496: - I spent some time trying to first-iteration-refactor the producer's sending activity, as a basis for playing around with adding some persistence to it I'll probably wait for this item here (KAFKA-496) to be implemented first. I'm writing here just to add that it would potentially be nice if the producer's wrappers around sending messages would become simplified in the code through some behavior-maintaining refactoring. The producer internals around managing the queue and around sync/async flows can probably be made much simpler in terms of class and method relationships (or the relationships between producer and producer.async), as part of modifications implied on the previous posts above. I'm writing this as it may seem that KAFKA-496 here may take care of refactoring in this area anyway. It seems this can help future modifications... > high level producer send should return a response > - > > Key: KAFKA-496 > URL: https://issues.apache.org/jira/browse/KAFKA-496 > Project: Kafka > Issue Type: Bug > Components: core >Reporter: Jun Rao >Assignee: Jay Kreps >Priority: Blocker > Labels: features > Fix For: 0.8.1 > > Original Estimate: 72h > Remaining Estimate: 72h > > Currently, Producer.send() doesn't return any value. In 0.8, since each > produce request will be acked, we should pass the response back. What we can > do is that if the producer is in sync mode, we can return a map of > (topic,partitionId) -> (errorcode, offset). If the producer is in async mode, > we can just return a null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-789) Producer-side persistence for delivery guarantee
Matan Safriel created KAFKA-789: --- Summary: Producer-side persistence for delivery guarantee Key: KAFKA-789 URL: https://issues.apache.org/jira/browse/KAFKA-789 Project: Kafka Issue Type: Improvement Components: producer Reporter: Matan Safriel Assignee: Jun Rao Priority: Minor Fix For: 0.9 A suggestion for higher guarantee for the part of entering messages into Kafka through it's producer. It aims to address the case that the entire set of broker replicas for a topic and partition is not available. Currently, in that case, data is lost. When a message set exhausts the send retry counter, the message set will be simply dropped. It would be nice being able to provide higher guarantee that a message passed to the producer would eventually be received by the broker. In an environment with some disk space to spare for this on the producer side, persisting to disk would seem to enable keeping messages for later retry (until defined space limits are exhausted). Thus somewhat elevating the level of guarantee. One way to facilitate this would be capitalizing on https://issues.apache.org/jira/browse/KAFKA-496, as the feedback it will add will enable knowing what needs to be retried again later. Changes to the producer or a wrapper around it (that may require access to the partitioning functions) would be able to persist failed message sets and manage delivery with a nice level of guarantee. As it would affect performance and use disks, should probably be a non-default option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-797) Cannot start Kafka 0.8 per Quick Start instructions
[ https://issues.apache.org/jira/browse/KAFKA-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597875#comment-13597875 ] Matan Safriel commented on KAFKA-797: - Back to the original topic, David, the problem you encountered (Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject) may seem to be related to Java not being able to run Scala on your machine. Can you share the versions of SBT and Java (java -version)? Also you may want to try in ./sbt: clean ++ 2.9.2 package HTH > Cannot start Kafka 0.8 per Quick Start instructions > --- > > Key: KAFKA-797 > URL: https://issues.apache.org/jira/browse/KAFKA-797 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: David Arthur > > Checked out latest 0.8 (b5edbb193b33ebf0b3056935a537967ff21478a6), and ran: > ./sbt update > ./sbt package > Both complete successfully. Then I try to start the server: > $ ./bin/kafka-server-start.sh config/server.properties > Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at kafka.Kafka.main(Kafka.scala) > Caused by: java.lang.ClassNotFoundException: scala.ScalaObject > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 25 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-156) Messages should not be dropped when brokers are unavailable
[ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598000#comment-13598000 ] Matan Safriel edited comment on KAFKA-156 at 3/9/13 5:17 PM: - I was led to believe this is most probably not planned for 0.8 now. Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well... was (Author: matan): I was led to believe this is most probably not planned for 0.8. Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well... > Messages should not be dropped when brokers are unavailable > --- > > Key: KAFKA-156 > URL: https://issues.apache.org/jira/browse/KAFKA-156 > Project: Kafka > Issue Type: Improvement >Reporter: Sharad Agarwal > Fix For: 0.8 > > > When none of the broker is available, producer should spool the messages to > disk and keep retrying for brokers to come back. > This will also enable brokers upgrade/maintenance without message loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-156) Messages should not be dropped when brokers are unavailable
[ https://issues.apache.org/jira/browse/KAFKA-156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598000#comment-13598000 ] Matan Safriel commented on KAFKA-156: - I was led to believe this is most probably not planned for 0.8. Just to comment that currently in 0.8 a message set asynchronously sent may partially fail, in the sense that only some of its destinations (determined by topic & partition) will fail while others succeed. That seems to mean a 'one log spool' for all messages (if all messages are spooled before attempting to send them) requires non-sequential management of such a log, or overheads over-proportional to the proportion of failed messages. Perhaps a log per topic would simplify the algorithm but then there's partitions as well... > Messages should not be dropped when brokers are unavailable > --- > > Key: KAFKA-156 > URL: https://issues.apache.org/jira/browse/KAFKA-156 > Project: Kafka > Issue Type: Improvement >Reporter: Sharad Agarwal > Fix For: 0.8 > > > When none of the broker is available, producer should spool the messages to > disk and keep retrying for brokers to come back. > This will also enable brokers upgrade/maintenance without message loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
