Joe, I there also a way to generate a sources jar via sbt?
Thanks, Jason On Mon, Jun 17, 2013 at 5:28 PM, Jason Rosenberg <j...@squareup.com> wrote: > Another thing I notice, is that the sample 'config/server.properties' in > many cases seems to have no correlation to the default config properties. > In some cases, it offers much greater allocation (e.g. for > socket.send.buffer.bytes, and for others much less. > > Have all the sample settings in config/server.properties been vetted for > the new config values? > > Jason > > > On Mon, Jun 17, 2013 at 5:23 PM, Joe Stein <crypt...@gmail.com> wrote: > >> you can use this to build the binary distributable >> >> ./sbt release-tar >> >> >> >> >> On Mon, Jun 17, 2013 at 8:17 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >> >> > Looking at the convenience script for running kafka server, it builds a >> > classpath by looking in several directories (looking at >> > kafka-run-class.sh). >> > >> > This script includes a jar under perf/target, should I care about that, >> > when launching a mavenized version of the server? >> > >> > Also, it includes jars under the "libs/" directory, which doesn't exist >> > (there is a "lib/') dir. Is that an oversight? >> > >> > Finally, it includes top-level jars that match kafka*.jar, which don't >> > exist from what I can tell. >> > >> > Or is there an extra sbt command I need to run to build out more >> > directories? >> > >> > Jason >> > >> > >> > On Mon, Jun 17, 2013 at 4:48 PM, Jason Rosenberg <j...@squareup.com> >> wrote: >> > >> > > Ok, >> > > >> > > So it seems the issue is related somehow to how I've wrapped the >> server >> > > (using a container app, using the maven pom from "./sbt make-pom"). >> If >> > I >> > > start the server using the script "kafka-server-start.sh", it works >> fine. >> > > Still looking (but haven't been able to see anything obvious so far). >> > > >> > > Jason >> > > >> > > >> > > On Sun, Jun 16, 2013 at 8:46 PM, Joe Stein <crypt...@gmail.com> >> wrote: >> > > >> > >> Jason, >> > >> >> > >> I just uploaded a signed binary build (kafka_2.8.2-0.8.0-beta1.jar) >> to >> > the >> > >> artifacts >> > >> http://people.apache.org/~joestein/kafka-0.8.0-beta1-candidate1/ >> > >> >> > >> I also uploaded a 2.9.2 binary too kafka_2.9.2-0.8.0-beta1.jar >> > >> >> > >> >> > >> On Sun, Jun 16, 2013 at 11:36 PM, Jason Rosenberg <j...@squareup.com> >> > >> wrote: >> > >> >> > >> > Joe, >> > >> > >> > >> > So I am using the 2.8.2 build of the kafka jar, using that latest >> > >> > beta1-candidate1 tag. >> > >> > >> > >> > The code above should be all you need to reproduce the issue. I'll >> > >> create >> > >> > a JIRA ticket. >> > >> > >> > >> > Thanks, >> > >> > >> > >> > Jason >> > >> > >> > >> > >> > >> > On Sun, Jun 16, 2013 at 8:32 PM, Joe Stein <crypt...@gmail.com> >> > wrote: >> > >> > >> > >> > > I have been running tests on 2.9.2 and 2.8.2 without any issues >> off >> > >> the >> > >> > > beta1-candidate1 release tag >> > >> > > >> > >> > > >> > >> > >> > >> >> > >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=refs/tags/0.8.0-beta1 >> > >> > > >> > >> > > Binaries will be built from the source file available from >> download >> > >> here >> > >> > : >> > >> > > http://people.apache.org/~joestein/kafka-0.8.0-beta1-candidate1/ >> > >> > > >> > >> > > It could be that test failing is interacting with a local file >> > already >> > >> > > there and causing issues or such type of thing not reproducible >> > maybe >> > >> > > without failing the test first to cause the data not cleaned up >> or >> > >> > > something, dunno. >> > >> > > >> > >> > > If you have an issue can you put the steps to reproduce it >> together >> > or >> > >> > > whatever amount of detailed information you can provide and then >> > >> please >> > >> > > open up a JIRA ticket >> https://issues.apache.org/jira/browse/KAFKA >> > >> > > >> > >> > > Thanks! >> > >> > > >> > >> > > >> > >> > > On Sun, Jun 16, 2013 at 11:14 PM, Jason Rosenberg < >> j...@squareup.com >> > > >> > >> > > wrote: >> > >> > > >> > >> > > > Yep, >> > >> > > > >> > >> > > > The configs are good. And my apps are working fine too. It >> seems >> > >> only >> > >> > > to >> > >> > > > be an issue with this test (and a few others like it). >> > >> > > > >> > >> > > > Jason >> > >> > > > >> > >> > > > >> > >> > > > On Sun, Jun 16, 2013 at 3:02 PM, Eric Sites < >> > >> > eric.si...@threattrack.com >> > >> > > > >wrote: >> > >> > > > >> > >> > > > > Jason, >> > >> > > > > >> > >> > > > > >> > >> > > > > Did you update your config file with the new name of the >> > zookeeper >> > >> > > > > settings: >> > >> > > > > >> > >> > > > > It was renamed from zk.connect to zookeeper.connect. >> > >> > > > > >> > >> > > > > You should check all of the settings because other setting >> names >> > >> have >> > >> > > > > changed as well. >> > >> > > > > >> > >> > > > > Cheers, >> > >> > > > > Eric Sites >> > >> > > > > >> > >> > > > > On 6/16/13 5:14 PM, "Jason Rosenberg" <j...@squareup.com> >> wrote: >> > >> > > > > >> > >> > > > > >I've started having problems with the latest version of the >> 0.8 >> > >> > > branch. >> > >> > > > > > The test below has started failing. It was working fine >> with >> > a >> > >> > prior >> > >> > > > > >version of 0.8, going back to Apr 30 >> > >> > > > > >(sha 988d4d8e65a14390abd748318a64e281e4a37c19). >> > >> > > > > > >> > >> > > > > >I haven't figured out when exactly it started failing, but I >> > saw >> > >> it >> > >> > > > with a >> > >> > > > > >version on Jun 9 (sha >> ddb7947c05583ea317e8f994f07b83bf6d5213c3) >> > >> and >> > >> > > now >> > >> > > > > >also with the latest (sha >> > >> 23acbd309f5e17de71db46cb6f1a60c8d38ea4e4). >> > >> > > > > > >> > >> > > > > >The test code is essentially this (assume a zk server is >> > running >> > >> > with >> > >> > > > > >'zkConnect', and a kafka broker running with a metadata >> port at >> > >> > > 'port': >> > >> > > > > > >> > >> > > > > > Properties pProps = new Properties(); >> > >> > > > > > pProps.put("metadata.broker.list", "localhost:" + port); >> > >> > > > > > pProps.put("serializer.class", >> > >> > "kafka.serializer.StringEncoder"); >> > >> > > > > > ProducerConfig pConfig = new ProducerConfig(pProps); >> > >> > > > > > Producer<Integer, String> producer = new >> Producer<Integer, >> > >> > > > > >String>(pConfig); >> > >> > > > > > KeyedMessage<Integer, String> data = >> > >> > > > > > new KeyedMessage<Integer, String>("test-topic", >> > >> > > "test-message"); >> > >> > > > > > producer.send(data); >> > >> > > > > > producer.close(); >> > >> > > > > > >> > >> > > > > > Properties cProps = new Properties(); >> > >> > > > > > cProps.put("zookeeper.connect", zkConnect); >> > >> > > > > > cProps.put("group.id", "group1"); >> > >> > > > > > ConsumerConfig consumerConfig = new >> ConsumerConfig(cProps); >> > >> > > > > > ConsumerConnector consumerConnector = >> > >> > > > > >Consumer.createJavaConsumerConnector(consumerConfig); >> > >> > > > > > >> > >> > > > > > Map<String, List<KafkaStream<byte[], byte[]>>> >> > >> > > topicMessageStreams = >> > >> > > > > > >> > >> > > > > >> > >> >consumerConnector.createMessageStreams(ImmutableMap.of("test-topic", >> > >> > > > 1)); >> > >> > > > > > List<KafkaStream<byte[], byte[]>> streams = >> > >> > > > > >topicMessageStreams.get("test-topic"); >> > >> > > > > > final KafkaStream<byte[], byte[]> stream = >> streams.get(0); >> > >> > > > > > final ConsumerIterator<byte[], byte[]> iter = >> > >> stream.iterator(); >> > >> > > > > > >> > >> > > > > > // run in a separate thread >> > >> > > > > > final AtomicBoolean success = new AtomicBoolean(false); >> > >> > > > > > Thread consumeThread = new Thread(new Runnable() { >> > >> > > > > > public void run() { >> > >> > > > > > while (iter.hasNext()) { >> > >> > > > > > byte[] msg = iter.next().message(); >> > >> > > > > > String msgStr = new String(msg); >> > >> > > > > > success.set(msgStr.equals("test-message")); >> > >> > > > > > break; >> > >> > > > > > } >> > >> > > > > > } >> > >> > > > > > }); >> > >> > > > > > >> > >> > > > > > consumeThread.start(); >> > >> > > > > > // this now hangs with the latest code >> > >> > > > > > consumeThread.join(); >> > >> > > > > > >> > >> > > > > > consumerConnector.shutdown(); >> > >> > > > > > assertTrue(success.get()); >> > >> > > > > > >> > >> > > > > >The output looks like this: >> > >> > > > > > >> > >> > > > > >912 [main] WARN kafka.producer.BrokerPartitionInfo - Error >> > while >> > >> > > > fetching >> > >> > > > > >metadata [{TopicMetadata for topic test-topic -> >> > >> > > > > >No partition metadata for topic test-topic due to >> > >> > > > > >kafka.common.LeaderNotAvailableException}] for topic >> > >> [test-topic]: >> > >> > > class >> > >> > > > > >kafka.common.LeaderNotAvailableException >> > >> > > > > >922 [main] WARN kafka.producer.BrokerPartitionInfo - Error >> > while >> > >> > > > fetching >> > >> > > > > >metadata [{TopicMetadata for topic test-topic -> >> > >> > > > > >No partition metadata for topic test-topic due to >> > >> > > > > >kafka.common.LeaderNotAvailableException}] for topic >> > >> [test-topic]: >> > >> > > class >> > >> > > > > >kafka.common.LeaderNotAvailableException >> > >> > > > > >923 [main] ERROR kafka.producer.async.DefaultEventHandler - >> > >> Failed >> > >> > to >> > >> > > > > >collate messages by topic, partition due to: Failed to fetch >> > >> topic >> > >> > > > > >metadata >> > >> > > > > >for topic: test-topic >> > >> > > > > >980 [kafka-request-handler-2] WARN >> > >> > > kafka.server.HighwaterMarkCheckpoint >> > >> > > > - >> > >> > > > > >No highwatermark file is found. Returning 0 as the >> > highwatermark >> > >> for >> > >> > > > > >partition [test-topic,0] >> > >> > > > > > >> > >> > > > > >The consumer never receives a message, and so the test >> > hangs.... >> > >> > > > > > >> > >> > > > > >This test worked fine as I said with an older version of the >> > >> branch, >> > >> > > but >> > >> > > > > >it >> > >> > > > > >would output exceptions about LeaderNotAvailable, etc... >> > >> > > > > > >> > >> > > > > >Thoughts? >> > >> > > > > > >> > >> > > > > >Jason >> > >> > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > -- >> > >> > > >> > >> > > /* >> > >> > > Joe Stein >> > >> > > http://www.linkedin.com/in/charmalloc >> > >> > > Twitter: @allthingshadoop < >> http://www.twitter.com/allthingshadoop> >> > >> > > */ >> > >> > > >> > >> > >> > >> >> > >> >> > >> >> > >> -- >> > >> >> > >> /* >> > >> Joe Stein >> > >> http://www.linkedin.com/in/charmalloc >> > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> > >> */ >> > >> >> > > >> > > >> > >> >> >> >> -- >> >> /* >> Joe Stein >> http://www.linkedin.com/in/charmalloc >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> */ >> > >
