Ok, Contrary to what I said above, the issue does seem to occur when kafka is launched by the sample startup scripts. I've filed a bug, with a test-case: https://issues.apache.org/jira/browse/KAFKA-945
I'm hoping there's a simple thing to change in the test code that will solve this (but I suspect there's an underlying problem). Jason On Mon, Jun 17, 2013 at 6:05 PM, Jason Rosenberg <j...@squareup.com> wrote: > Hmmm....that's not working for me (no *.sources.jar files are generated). > do I need to add a flag? > > > On Mon, Jun 17, 2013 at 5:47 PM, Joe Stein <crypt...@gmail.com> wrote: > >> Yup, depending on your Scala version >> >> ./sbt "++2.9.2 package" >> >> 2.8.0, 2.8.2, 2.9.1, 2.9.2 are available >> >> /* >> Joe Stein, Chief Architect >> http://www.medialets.com >> Twitter: @allthingshadoop >> Mobile: 917-597-9771 >> */ >> >> On Jun 17, 2013, at 8:41 PM, Jason Rosenberg <j...@squareup.com> wrote: >> >> > Joe, >> > >> > I there also a way to generate a sources jar via sbt? >> > >> > Thanks, >> > >> > Jason >> > >> > >> > On Mon, Jun 17, 2013 at 5:28 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >> > >> >> Another thing I notice, is that the sample 'config/server.properties' >> in >> >> many cases seems to have no correlation to the default config >> properties. >> >> In some cases, it offers much greater allocation (e.g. for >> >> socket.send.buffer.bytes, and for others much less. >> >> >> >> Have all the sample settings in config/server.properties been vetted >> for >> >> the new config values? >> >> >> >> Jason >> >> >> >> >> >> On Mon, Jun 17, 2013 at 5:23 PM, Joe Stein <crypt...@gmail.com> wrote: >> >> >> >>> you can use this to build the binary distributable >> >>> >> >>> ./sbt release-tar >> >>> >> >>> >> >>> >> >>> >> >>> On Mon, Jun 17, 2013 at 8:17 PM, Jason Rosenberg <j...@squareup.com> >> >>> wrote: >> >>> >> >>>> Looking at the convenience script for running kafka server, it >> builds a >> >>>> classpath by looking in several directories (looking at >> >>>> kafka-run-class.sh). >> >>>> >> >>>> This script includes a jar under perf/target, should I care about >> that, >> >>>> when launching a mavenized version of the server? >> >>>> >> >>>> Also, it includes jars under the "libs/" directory, which doesn't >> exist >> >>>> (there is a "lib/') dir. Is that an oversight? >> >>>> >> >>>> Finally, it includes top-level jars that match kafka*.jar, which >> don't >> >>>> exist from what I can tell. >> >>>> >> >>>> Or is there an extra sbt command I need to run to build out more >> >>>> directories? >> >>>> >> >>>> Jason >> >>>> >> >>>> >> >>>> On Mon, Jun 17, 2013 at 4:48 PM, Jason Rosenberg <j...@squareup.com> >> >>> wrote: >> >>>> >> >>>>> Ok, >> >>>>> >> >>>>> So it seems the issue is related somehow to how I've wrapped the >> >>> server >> >>>>> (using a container app, using the maven pom from "./sbt make-pom"). >> >>> If >> >>>> I >> >>>>> start the server using the script "kafka-server-start.sh", it works >> >>> fine. >> >>>>> Still looking (but haven't been able to see anything obvious so >> far). >> >>>>> >> >>>>> Jason >> >>>>> >> >>>>> >> >>>>> On Sun, Jun 16, 2013 at 8:46 PM, Joe Stein <crypt...@gmail.com> >> >>> wrote: >> >>>>> >> >>>>>> Jason, >> >>>>>> >> >>>>>> I just uploaded a signed binary build (kafka_2.8.2-0.8.0-beta1.jar) >> >>> to >> >>>> the >> >>>>>> artifacts >> >>>>>> http://people.apache.org/~joestein/kafka-0.8.0-beta1-candidate1/ >> >>>>>> >> >>>>>> I also uploaded a 2.9.2 binary too kafka_2.9.2-0.8.0-beta1.jar >> >>>>>> >> >>>>>> >> >>>>>> On Sun, Jun 16, 2013 at 11:36 PM, Jason Rosenberg < >> j...@squareup.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> Joe, >> >>>>>>> >> >>>>>>> So I am using the 2.8.2 build of the kafka jar, using that latest >> >>>>>>> beta1-candidate1 tag. >> >>>>>>> >> >>>>>>> The code above should be all you need to reproduce the issue. >> I'll >> >>>>>> create >> >>>>>>> a JIRA ticket. >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> Jason >> >>>>>>> >> >>>>>>> >> >>>>>>> On Sun, Jun 16, 2013 at 8:32 PM, Joe Stein <crypt...@gmail.com> >> >>>> wrote: >> >>>>>>> >> >>>>>>>> I have been running tests on 2.9.2 and 2.8.2 without any issues >> >>> off >> >>>>>> the >> >>>>>>>> beta1-candidate1 release tag >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>> >> >>> >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=refs/tags/0.8.0-beta1 >> >>>>>>>> >> >>>>>>>> Binaries will be built from the source file available from >> >>> download >> >>>>>> here >> >>>>>>> : >> >>>>>>>> http://people.apache.org/~joestein/kafka-0.8.0-beta1-candidate1/ >> >>>>>>>> >> >>>>>>>> It could be that test failing is interacting with a local file >> >>>> already >> >>>>>>>> there and causing issues or such type of thing not reproducible >> >>>> maybe >> >>>>>>>> without failing the test first to cause the data not cleaned up >> >>> or >> >>>>>>>> something, dunno. >> >>>>>>>> >> >>>>>>>> If you have an issue can you put the steps to reproduce it >> >>> together >> >>>> or >> >>>>>>>> whatever amount of detailed information you can provide and then >> >>>>>> please >> >>>>>>>> open up a JIRA ticket >> >>> https://issues.apache.org/jira/browse/KAFKA >> >>>>>>>> >> >>>>>>>> Thanks! >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Sun, Jun 16, 2013 at 11:14 PM, Jason Rosenberg < >> >>> j...@squareup.com >> >>>>> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Yep, >> >>>>>>>>> >> >>>>>>>>> The configs are good. And my apps are working fine too. It >> >>> seems >> >>>>>> only >> >>>>>>>> to >> >>>>>>>>> be an issue with this test (and a few others like it). >> >>>>>>>>> >> >>>>>>>>> Jason >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Sun, Jun 16, 2013 at 3:02 PM, Eric Sites < >> >>>>>>> eric.si...@threattrack.com >> >>>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Jason, >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Did you update your config file with the new name of the >> >>>> zookeeper >> >>>>>>>>>> settings: >> >>>>>>>>>> >> >>>>>>>>>> It was renamed from zk.connect to zookeeper.connect. >> >>>>>>>>>> >> >>>>>>>>>> You should check all of the settings because other setting >> >>> names >> >>>>>> have >> >>>>>>>>>> changed as well. >> >>>>>>>>>> >> >>>>>>>>>> Cheers, >> >>>>>>>>>> Eric Sites >> >>>>>>>>>> >> >>>>>>>>>> On 6/16/13 5:14 PM, "Jason Rosenberg" <j...@squareup.com> >> >>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> I've started having problems with the latest version of the >> >>> 0.8 >> >>>>>>>> branch. >> >>>>>>>>>>> The test below has started failing. It was working fine >> >>> with >> >>>> a >> >>>>>>> prior >> >>>>>>>>>>> version of 0.8, going back to Apr 30 >> >>>>>>>>>>> (sha 988d4d8e65a14390abd748318a64e281e4a37c19). >> >>>>>>>>>>> >> >>>>>>>>>>> I haven't figured out when exactly it started failing, but I >> >>>> saw >> >>>>>> it >> >>>>>>>>> with a >> >>>>>>>>>>> version on Jun 9 (sha >> >>> ddb7947c05583ea317e8f994f07b83bf6d5213c3) >> >>>>>> and >> >>>>>>>> now >> >>>>>>>>>>> also with the latest (sha >> >>>>>> 23acbd309f5e17de71db46cb6f1a60c8d38ea4e4). >> >>>>>>>>>>> >> >>>>>>>>>>> The test code is essentially this (assume a zk server is >> >>>> running >> >>>>>>> with >> >>>>>>>>>>> 'zkConnect', and a kafka broker running with a metadata >> >>> port at >> >>>>>>>> 'port': >> >>>>>>>>>>> >> >>>>>>>>>>> Properties pProps = new Properties(); >> >>>>>>>>>>> pProps.put("metadata.broker.list", "localhost:" + port); >> >>>>>>>>>>> pProps.put("serializer.class", >> >>>>>>> "kafka.serializer.StringEncoder"); >> >>>>>>>>>>> ProducerConfig pConfig = new ProducerConfig(pProps); >> >>>>>>>>>>> Producer<Integer, String> producer = new >> >>> Producer<Integer, >> >>>>>>>>>>> String>(pConfig); >> >>>>>>>>>>> KeyedMessage<Integer, String> data = >> >>>>>>>>>>> new KeyedMessage<Integer, String>("test-topic", >> >>>>>>>> "test-message"); >> >>>>>>>>>>> producer.send(data); >> >>>>>>>>>>> producer.close(); >> >>>>>>>>>>> >> >>>>>>>>>>> Properties cProps = new Properties(); >> >>>>>>>>>>> cProps.put("zookeeper.connect", zkConnect); >> >>>>>>>>>>> cProps.put("group.id", "group1"); >> >>>>>>>>>>> ConsumerConfig consumerConfig = new >> >>> ConsumerConfig(cProps); >> >>>>>>>>>>> ConsumerConnector consumerConnector = >> >>>>>>>>>>> Consumer.createJavaConsumerConnector(consumerConfig); >> >>>>>>>>>>> >> >>>>>>>>>>> Map<String, List<KafkaStream<byte[], byte[]>>> >> >>>>>>>> topicMessageStreams = >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>> >> consumerConnector.createMessageStreams(ImmutableMap.of("test-topic", >> >>>>>>>>> 1)); >> >>>>>>>>>>> List<KafkaStream<byte[], byte[]>> streams = >> >>>>>>>>>>> topicMessageStreams.get("test-topic"); >> >>>>>>>>>>> final KafkaStream<byte[], byte[]> stream = >> >>> streams.get(0); >> >>>>>>>>>>> final ConsumerIterator<byte[], byte[]> iter = >> >>>>>> stream.iterator(); >> >>>>>>>>>>> >> >>>>>>>>>>> // run in a separate thread >> >>>>>>>>>>> final AtomicBoolean success = new AtomicBoolean(false); >> >>>>>>>>>>> Thread consumeThread = new Thread(new Runnable() { >> >>>>>>>>>>> public void run() { >> >>>>>>>>>>> while (iter.hasNext()) { >> >>>>>>>>>>> byte[] msg = iter.next().message(); >> >>>>>>>>>>> String msgStr = new String(msg); >> >>>>>>>>>>> success.set(msgStr.equals("test-message")); >> >>>>>>>>>>> break; >> >>>>>>>>>>> } >> >>>>>>>>>>> } >> >>>>>>>>>>> }); >> >>>>>>>>>>> >> >>>>>>>>>>> consumeThread.start(); >> >>>>>>>>>>> // this now hangs with the latest code >> >>>>>>>>>>> consumeThread.join(); >> >>>>>>>>>>> >> >>>>>>>>>>> consumerConnector.shutdown(); >> >>>>>>>>>>> assertTrue(success.get()); >> >>>>>>>>>>> >> >>>>>>>>>>> The output looks like this: >> >>>>>>>>>>> >> >>>>>>>>>>> 912 [main] WARN kafka.producer.BrokerPartitionInfo - Error >> >>>> while >> >>>>>>>>> fetching >> >>>>>>>>>>> metadata [{TopicMetadata for topic test-topic -> >> >>>>>>>>>>> No partition metadata for topic test-topic due to >> >>>>>>>>>>> kafka.common.LeaderNotAvailableException}] for topic >> >>>>>> [test-topic]: >> >>>>>>>> class >> >>>>>>>>>>> kafka.common.LeaderNotAvailableException >> >>>>>>>>>>> 922 [main] WARN kafka.producer.BrokerPartitionInfo - Error >> >>>> while >> >>>>>>>>> fetching >> >>>>>>>>>>> metadata [{TopicMetadata for topic test-topic -> >> >>>>>>>>>>> No partition metadata for topic test-topic due to >> >>>>>>>>>>> kafka.common.LeaderNotAvailableException}] for topic >> >>>>>> [test-topic]: >> >>>>>>>> class >> >>>>>>>>>>> kafka.common.LeaderNotAvailableException >> >>>>>>>>>>> 923 [main] ERROR kafka.producer.async.DefaultEventHandler - >> >>>>>> Failed >> >>>>>>> to >> >>>>>>>>>>> collate messages by topic, partition due to: Failed to fetch >> >>>>>> topic >> >>>>>>>>>>> metadata >> >>>>>>>>>>> for topic: test-topic >> >>>>>>>>>>> 980 [kafka-request-handler-2] WARN >> >>>>>>>> kafka.server.HighwaterMarkCheckpoint >> >>>>>>>>> - >> >>>>>>>>>>> No highwatermark file is found. Returning 0 as the >> >>>> highwatermark >> >>>>>> for >> >>>>>>>>>>> partition [test-topic,0] >> >>>>>>>>>>> >> >>>>>>>>>>> The consumer never receives a message, and so the test >> >>>> hangs.... >> >>>>>>>>>>> >> >>>>>>>>>>> This test worked fine as I said with an older version of the >> >>>>>> branch, >> >>>>>>>> but >> >>>>>>>>>>> it >> >>>>>>>>>>> would output exceptions about LeaderNotAvailable, etc... >> >>>>>>>>>>> >> >>>>>>>>>>> Thoughts? >> >>>>>>>>>>> >> >>>>>>>>>>> Jason >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> >> >>>>>>>> /* >> >>>>>>>> Joe Stein >> >>>>>>>> http://www.linkedin.com/in/charmalloc >> >>>>>>>> Twitter: @allthingshadoop < >> >>> http://www.twitter.com/allthingshadoop> >> >>>>>>>> */ >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> /* >> >>>>>> Joe Stein >> >>>>>> http://www.linkedin.com/in/charmalloc >> >>>>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> >>>>>> */ >> >>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> >> >>> -- >> >>> >> >>> /* >> >>> Joe Stein >> >>> http://www.linkedin.com/in/charmalloc >> >>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> >>> */ >> >>> >> >> >> >> >> > >
