[jira] [Resolved] (KAFKA-397) kafka.common.InvalidMessageSizeException: null

2013-07-12 Thread David Siegel (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Siegel resolved KAFKA-397.


Resolution: Not A Problem

> kafka.common.InvalidMessageSizeException: null
> --
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: David Siegel
>Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.  
> I made a backup of the kafka log directory and wiped the logs.  I restarting 
> kafka and the consumer.  After processing a few hundred messages successfully 
> I got the same error again.  I restarted the consumer again and got the same 
> error immediately.  I wiped the logs yet again and reproduced the error 
> again.  I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
>   producer config:
> max.message.size: 100
>   consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper 
> offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] 
> c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method) ~[na:1.6.0_30]
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  ~[na:1.6.0_30]
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) 
> ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at 
> kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
>  ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) 
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] 
> c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state.  Thread 
> exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) 
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-397) kafka.common.InvalidMessageSizeException: null

2013-07-12 Thread David Siegel (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706974#comment-13706974
 ] 

David Siegel commented on KAFKA-397:


I'm pretty sure this was my fault not clearing zookeeper properly.

> kafka.common.InvalidMessageSizeException: null
> --
>
> Key: KAFKA-397
> URL: https://issues.apache.org/jira/browse/KAFKA-397
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: David Siegel
>Priority: Blocker
> Attachments: kafka-logs-invalid-message-size-null.tar
>
>
> I've just gotten the following error while running the zookeeper consumer.  
> I made a backup of the kafka log directory and wiped the logs.  I restarting 
> kafka and the consumer.  After processing a few hundred messages successfully 
> I got the same error again.  I restarted the consumer again and got the same 
> error immediately.  I wiped the logs yet again and reproduced the error 
> again.  I will attach the logs from this final run.
> I'm running Kafka 0.7.1
> I have the following message size configurations:
>   producer config:
> max.message.size: 100
>   consumer config:
> fetch.size: 2072000
> Does it matter that I'm only wiping the logs and not wiping the zookeeper 
> offsets?
> 2012-07-10 02:31:21,998 ERROR [Consumer1] 
> c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event
> kafka.common.InvalidMessageSizeException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method) ~[na:1.6.0_30]
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  ~[na:1.6.0_30]
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  ~[na:1.6.0_30]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) 
> ~[na:1.6.0_30]
> at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30]
> at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30]
> at 
> kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
>  ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) 
> ~[KPIP-0.4.birdy.jar:na]
> at 
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) 
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]
> 2012-07-10 02:31:21,998 ERROR [Consumer1] 
> c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state.  Thread 
> exiting
> java.lang.IllegalStateException: Iterator is in failed state
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) 
> ~[KPIP-0.4.birdy.jar:na]
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) 
> ~[KPIP-0.4.birdy.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Kafka User Group Meeting Jun. 27

2013-07-12 Thread Jun Rao 
The link to the video recording of the event is posted in the meetup link (
http://www.meetup.com/http-kafka
-apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event).

Thanks,

Jun


On Tue, Jul 2, 2013 at 12:49 PM, Jay Kreps  wrote:

> Erp, actually I don't see a way to get the recording there, nm.
>
> -Jay
>
>
> On Tue, Jul 2, 2013 at 12:47 PM, Jay Kreps  wrote:
>
> > The recording for the user group talks is available here:
> > http://www.ustream.tv/linkedin-events
> >
> > -Jay
> >
> >
> > On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao  wrote:
> >
> >> Hi, Everyone,
> >>
> >> We have finalized our agenda of the meetup Thursday evening, with
> Speakers
> >> from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the
> >> meetup
> >> link below. For remote people, we will post the streaming link at meetup
> >> before the meeting and we will use our IRC for remote questions.
> >>
> >>
> >>
> http://www.meetup.com/http-kafka-apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event
> >>
> >> See you Thursday evening.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >
> >
>

[jira] [Updated] (KAFKA-347) change number of partitions of a topic online

2013-07-12 Thread Sriram Subramanian (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriram Subramanian updated KAFKA-347:
-

Status: Patch Available  (was: Open)

1. Added an isolated change to dynamically add partitions for messages without 
key
2. The only common code change is the replica layout strategy
3. Unit tests

> change number of partitions of a topic online
> -
>
> Key: KAFKA-347
> URL: https://issues.apache.org/jira/browse/KAFKA-347
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Sriram Subramanian
>  Labels: features
> Fix For: 0.8.1
>
> Attachments: kafka-347.patch
>
>
> We will need an admin tool to change the number of partitions of a topic 
> online.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-347) change number of partitions of a topic online

2013-07-12 Thread Sriram Subramanian (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriram Subramanian updated KAFKA-347:
-

Attachment: kafka-347.patch

> change number of partitions of a topic online
> -
>
> Key: KAFKA-347
> URL: https://issues.apache.org/jira/browse/KAFKA-347
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Sriram Subramanian
>  Labels: features
> Fix For: 0.8.1
>
> Attachments: kafka-347.patch
>
>
> We will need an admin tool to change the number of partitions of a topic 
> online.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Kafka User Group Meeting Jun. 27

2013-07-12 Thread S Ahmed 
thanks.

reference: http://www.youtube.com/watch?v=njuz0zSBvnc



On Fri, Jul 12, 2013 at 10:38 AM, Jun Rao  wrote:

> The link to the video recording of the event is posted in the meetup link (
> http://www.meetup.com/http-kafka
> -apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event).
>
> Thanks,
>
> Jun
>
>
> On Tue, Jul 2, 2013 at 12:49 PM, Jay Kreps  wrote:
>
> > Erp, actually I don't see a way to get the recording there, nm.
> >
> > -Jay
> >
> >
> > On Tue, Jul 2, 2013 at 12:47 PM, Jay Kreps  wrote:
> >
> > > The recording for the user group talks is available here:
> > > http://www.ustream.tv/linkedin-events
> > >
> > > -Jay
> > >
> > >
> > > On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao  wrote:
> > >
> > >> Hi, Everyone,
> > >>
> > >> We have finalized our agenda of the meetup Thursday evening, with
> > Speakers
> > >> from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the
> > >> meetup
> > >> link below. For remote people, we will post the streaming link at
> meetup
> > >> before the meeting and we will use our IRC for remote questions.
> > >>
> > >>
> > >>
> >
> http://www.meetup.com/http-kafka-apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event
> > >>
> > >> See you Thursday evening.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >
> > >
> >
>

Re: Running 0.8 against 0.7 data and zk

2013-07-12 Thread Jay Kreps 
That sounds like a a future frequently asked question. I wonder if we
should try to add some heuristic to check the format of the log and/or zk
to error out more gracefully?

Folks should we do this for 0.8?

-Jay


On Thu, Jul 11, 2013 at 10:42 PM, Jun Rao  wrote:

> No. If you point an 0.8 broker to a ZK with 0.7 layout, it will still hit
> some weird exception, maybe not exactly the same as specified in the jira.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jul 11, 2013 at 3:15 PM, Jay Kreps  wrote:
>
> > This JIRA has a patch but I thought we already did something like this:
> > https://issues.apache.org/jira/browse/KAFKA-686
> >
> > Am I just confused?
> >
> > -Jay
> >
>

[jira] [Commented] (KAFKA-347) change number of partitions of a topic online

2013-07-12 Thread Guozhang Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707434#comment-13707434
 ] 

Guozhang Wang commented on KAFKA-347:
-

A few comments:

1. In addPartitions function, we can set partitionStartIndex as the last 
partition of the partition instead of partition 0 of the topic, since by doing 
so as more and more of this function gets called, the broker owning partition 
0's leader will have more leader partitions assigned to it.

2. Some of the changes in ZookeeperConsumerConnector for KAFKA-969 is also 
included in the patch. Better trim them out for this patch.

3. Are there two getManualReplicaAssignment in AddPartitionsCommand and 
CreateTopicCommand with one parameter difference? If yes could we combine them 
into one?

Some minor stuff:

1. In AdminUtils, there are two more space indents for line of 
"ZkUtils.updatePersistentPath(zkClient, zkPath, jsonPartitionData)"

2. In createOrUpdateTopicPartitionAssignmentPathInZK, use

if {
  //
} else {
  //
}

> change number of partitions of a topic online
> -
>
> Key: KAFKA-347
> URL: https://issues.apache.org/jira/browse/KAFKA-347
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Sriram Subramanian
>  Labels: features
> Fix For: 0.8.1
>
> Attachments: kafka-347.patch
>
>
> We will need an admin tool to change the number of partitions of a topic 
> online.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-347) change number of partitions of a topic online

2013-07-12 Thread Sriram Subramanian (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707444#comment-13707444
 ] 

Sriram Subramanian commented on KAFKA-347:
--

1. That would not give you the right result. The start index is for the replica 
of the first partition we create. This is to continue from where we left for 
that topic.

2. That is on purpose. There is no change. Just if else fix.

3. This is on purpose to prevent any regression for 0.8.

> change number of partitions of a topic online
> -
>
> Key: KAFKA-347
> URL: https://issues.apache.org/jira/browse/KAFKA-347
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Sriram Subramanian
>  Labels: features
> Fix For: 0.8.1
>
> Attachments: kafka-347.patch
>
>
> We will need an admin tool to change the number of partitions of a topic 
> online.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (KAFKA-930) Integrate preferred replica election logic into kafka

2013-07-12 Thread Sriram Subramanian (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-930 started by Sriram Subramanian.

> Integrate preferred replica election logic into kafka
> -
>
> Key: KAFKA-930
> URL: https://issues.apache.org/jira/browse/KAFKA-930
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sriram Subramanian
>Assignee: Sriram Subramanian
> Fix For: 0.9
>
>
> It seems useful to integrate the preferred replica election logic into kafka 
> controller. A simple way to implement this would be to have a background 
> thread that periodically finds the topic partitions that are not assigned to 
> the preferred broker and initiate the move. We could come up with some 
> heuristics to initiate the move only if the imbalance over a specific 
> threshold in order to avoid rebalancing too aggressively. Making the software 
> do this reduces operational cost.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-973) Messages From Producer Not being Partitioned

2013-07-12 Thread Subbu Srinivasan (JIRA) 
Subbu Srinivasan created KAFKA-973:
--

 Summary: Messages From Producer Not being Partitioned 
 Key: KAFKA-973
 URL: https://issues.apache.org/jira/browse/KAFKA-973
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8
 Environment: Linux
Reporter: Subbu Srinivasan
Assignee: Neha Narkhede


I created a two node cluster.

2 zoo keepers
2 brokers
1 topic with replication factor (2) and no of partition 2.
my consumer group has two threads

1) From my Java client - I send few  messages to the topic. I have set multiple 
brokers
kafka2:9092,kafka1:9092.

Only one thread in my consumer always gets the messages. It looks like producer 
is not
partitioning the requests properly.

2) However if I send some sample using the simple console producer, I see 
multiple threads getting
requests and is load balanced.

What am I doing wrong in my client?


public class KafkaProducer {

  
  private final Properties props = new Properties();

  private static AtomicLong counter = new AtomicLong(0);
  kafka.javaapi.producer.Producer producer = null;
  
  public KafkaProducer() 
  {
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("metadata.broker.list", 
ConfigurationUtility.getKafkaHost());
producer = new kafka.javaapi.producer.Producer(new 
ProducerConfig(props));
  } 
  
  public void sendMessage(String msg) throws Exception
  {
  producer.send(new KeyedMessage(ConfigurationUtility.getTopicName(), msg));
  }   
  
  
  public static void main(String arg[]) throws Exception
  {
  
  ConfigurationUtility.setKafkaHost("kafka2:9092,kafka1:9092");
ConfigurationUtility.setTopicName("dnslog");

ConfigurationUtility.setZooKeeperHost("kafka1:2181,kafka2:2181");
ConfigurationUtility.setConsumerGroupId("dnslog");

for(int i = 0 ; i < 2 ; ++i)
{
(new 
KafkaProducer()).sendMessage(UUID.randomUUID().toString());
}
  }
}




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-974) can't use public release maven repo because of failure of downloaded dependency

2013-07-12 Thread Joe Stein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-974:


Summary: can't use public release maven repo because of failure of 
downloaded dependency  (was: can't build from public release maven repo because 
of failure of downloaded dependency)

> can't use public release maven repo because of failure of downloaded 
> dependency
> ---
>
> Key: KAFKA-974
> URL: https://issues.apache.org/jira/browse/KAFKA-974
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
>
> trying to use the 0.8.0-beta1 release from public maven
> name := "Stub"
> version := "1.0.0.0"
> scalaVersion := "2.9.2"
> mainClass := Some("Stub")
> libraryDependencies ++= Seq(
>   "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
> )
> results in
> Joes-MacBook-Air:stub joestein$ ./sbt compile
> [info] Set current project to default-63d5f2 (in build 
> file:/opt/medialets/SymanticManager/scala/stub/)
> [info] Updating 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2...
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
> [warn]  public: tried
> [warn]   
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [warn] :: problems summary ::
> [warn]  WARNINGS
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn] public: tried
> [warn]  http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar 
> (12ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> [error] 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2/*:update: 
> sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
> [error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-974) can't build from public release maven repo because of failure of downloaded dependency

2013-07-12 Thread Joe Stein (JIRA) 
Joe Stein created KAFKA-974:
---

 Summary: can't build from public release maven repo because of 
failure of downloaded dependency
 Key: KAFKA-974
 URL: https://issues.apache.org/jira/browse/KAFKA-974
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8


trying to use the 0.8.0-beta1 release from public maven


name := "Stub"

version := "1.0.0.0"

scalaVersion := "2.9.2"

mainClass := Some("Stub")

libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
)

results in

Joes-MacBook-Air:stub joestein$ ./sbt compile
[info] Set current project to default-63d5f2 (in build 
file:/opt/medialets/SymanticManager/scala/stub/)
[info] Updating 
{file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2...
[warn]  [NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
[warn]  public: tried
[warn]   http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
[warn]  [NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
[warn]  public: tried
[warn]   
http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
[warn]  [NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
[warn]  public: tried
[warn]   http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
[warn]  ::
[warn]  ::  FAILED DOWNLOADS::
[warn]  :: ^ see resolution messages for details  ^ ::
[warn]  ::
[warn]  :: javax.jms#jms;1.1!jms.jar
[warn]  :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
[warn]  :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
[warn]  ::
[info] 
[warn] :: problems summary ::
[warn]  WARNINGS
[warn]  [NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
[warn]   public: tried
[warn]http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
[warn]  [NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
[warn]   public: tried
[warn]
http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
[warn]  [NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
[warn]   public: tried
[warn]http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
[warn]  ::
[warn]  ::  FAILED DOWNLOADS::
[warn]  :: ^ see resolution messages for details  ^ ::
[warn]  ::
[warn]  :: javax.jms#jms;1.1!jms.jar
[warn]  :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
[warn]  :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
[warn]  ::
[info] 
[info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
[error] 
{file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2/*:update: 
sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
[error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
[error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
[error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-974) can't use public release maven repo because of failure of downloaded dependency

2013-07-12 Thread Joe Stein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-974:


Status: Patch Available  (was: Open)

> can't use public release maven repo because of failure of downloaded 
> dependency
> ---
>
> Key: KAFKA-974
> URL: https://issues.apache.org/jira/browse/KAFKA-974
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-974.patch
>
>
> trying to use the 0.8.0-beta1 release from public maven
> name := "Stub"
> version := "1.0.0.0"
> scalaVersion := "2.9.2"
> mainClass := Some("Stub")
> libraryDependencies ++= Seq(
>   "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
> )
> results in
> Joes-MacBook-Air:stub joestein$ ./sbt compile
> [info] Set current project to default-63d5f2 (in build 
> file:/opt/medialets/SymanticManager/scala/stub/)
> [info] Updating 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2...
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
> [warn]  public: tried
> [warn]   
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [warn] :: problems summary ::
> [warn]  WARNINGS
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn] public: tried
> [warn]  http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar 
> (12ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> [error] 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2/*:update: 
> sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
> [error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-974) can't use public release maven repo because of failure of downloaded dependency

2013-07-12 Thread Joe Stein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-974:


Attachment: KAFKA-974.patch

> can't use public release maven repo because of failure of downloaded 
> dependency
> ---
>
> Key: KAFKA-974
> URL: https://issues.apache.org/jira/browse/KAFKA-974
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-974.patch
>
>
> trying to use the 0.8.0-beta1 release from public maven
> name := "Stub"
> version := "1.0.0.0"
> scalaVersion := "2.9.2"
> mainClass := Some("Stub")
> libraryDependencies ++= Seq(
>   "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
> )
> results in
> Joes-MacBook-Air:stub joestein$ ./sbt compile
> [info] Set current project to default-63d5f2 (in build 
> file:/opt/medialets/SymanticManager/scala/stub/)
> [info] Updating 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2...
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
> [warn]  public: tried
> [warn]   
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [warn] :: problems summary ::
> [warn]  WARNINGS
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn] public: tried
> [warn]  http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar 
> (12ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> [error] 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2/*:update: 
> sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
> [error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-974) can't use public release maven repo because of failure of downloaded dependency

2013-07-12 Thread Joe Stein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-974:


Status: Open  (was: Patch Available)

first patch that didn't work

> can't use public release maven repo because of failure of downloaded 
> dependency
> ---
>
> Key: KAFKA-974
> URL: https://issues.apache.org/jira/browse/KAFKA-974
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-974.patch
>
>
> trying to use the 0.8.0-beta1 release from public maven
> name := "Stub"
> version := "1.0.0.0"
> scalaVersion := "2.9.2"
> mainClass := Some("Stub")
> libraryDependencies ++= Seq(
>   "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
> )
> results in
> Joes-MacBook-Air:stub joestein$ ./sbt compile
> [info] Set current project to default-63d5f2 (in build 
> file:/opt/medialets/SymanticManager/scala/stub/)
> [info] Updating 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2...
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar (12ms)
> [warn]  public: tried
> [warn]   
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn]  public: tried
> [warn]   http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [warn] :: problems summary ::
> [warn]  WARNINGS
> [warn][NOT FOUND  ] javax.jms#jms;1.1!jms.jar (50ms)
> [warn] public: tried
> [warn]  http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [warn][NOT FOUND  ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar 
> (12ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [warn][NOT FOUND  ] com.sun.jmx#jmxri;1.2.1!jmxri.jar (71ms)
> [warn] public: tried
> [warn]  
> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [warn]::
> [warn]::  FAILED DOWNLOADS::
> [warn]:: ^ see resolution messages for details  ^ ::
> [warn]::
> [warn]:: javax.jms#jms;1.1!jms.jar
> [warn]:: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [warn]:: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [warn]::
> [info] 
> [info] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> [error] 
> {file:/opt/medialets/SymanticManager/scala/stub/}default-63d5f2/*:update: 
> sbt.ResolveException: download failed: javax.jms#jms;1.1!jms.jar
> [error] download failed: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [error] download failed: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [error] Total time: 3 s, completed Jul 13, 2013 1:55:36 AM

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Kafka distribution on public Maven repo

2013-07-12 Thread Joe Stein 
There is an issue with the pom not having the dependency exclusion correct
(at least I believe that is the problem now being tackled)

I tried a fix but it did not work
https://issues.apache.org/jira/browse/KAFKA-974

if someone gets a chance to look at it or knows the fix before I get a
chance to look at it again please update the JIRA


On Fri, Jul 12, 2013 at 11:41 PM, chetan conikee  wrote:

> Hey Jay
>
> This question might have been posed a million times, but thought I might
> pose it again.
>
> Why has kafka not been published to a public Maven repo?
>
> Chetan
>



-- 

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop 
*/