Regarding more information: Maybe ltrace? If I were you, I'd go to MigrationTool code and start adding LOG lines. because there aren't enough of those to troubleshoot.
On Wed, Nov 5, 2014 at 6:13 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > org.apache.zookeeper.ClientCnxn is throwing the exception, so I'm 100% > sure it eventually found the class. > > On Wed, Nov 5, 2014 at 5:59 PM, Tomas Nunez <nu...@pythian.com> wrote: > >> Ok, still fighting with the migrationTool here... >> >> That tuple wasn't in the scala-library.jar. It turns out I was using scala >> 2.10 for kafka0.8 and scala 2.8 for kafka0.7, and the jar files were not >> compatible. So, for the record, it seems that you need both the 0.7 jar >> files and your 0.8 kafka compiled with the same java version. >> >> After fixing that (downloading kafka 0.8 compiled with scala 2.8), I'm now >> facing a different error, this time more crypic: >> >> Kafka migration tool failed due to: >> java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:204) >> Caused by: java.lang.NullPointerException >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:361) >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332) >> at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383) >> at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) >> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872) >> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) >> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84) >> at >> >> kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:152) >> at >> >> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:122) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:65) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:67) >> at >> >> kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:88) >> at >> >> kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala) >> ... 5 more >> >> [2014-11-06 01:32:44,362] ERROR Kafka migration tool failed: >> (kafka.tools.KafkaMigrationTool) >> java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:204) >> Caused by: java.lang.NullPointerException >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:361) >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332) >> at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383) >> at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64) >> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872) >> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) >> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84) >> at >> >> kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:152) >> at >> >> kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:122) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:65) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:67) >> at >> >> kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:88) >> at >> >> kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala) >> ... 5 more >> >> Using strace with it, I could see these errors again: >> >> 13342 stat("/var/migration/kafka/utils/Logging.class", 0x7fc92913c280) = >> -1 >> ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/common/KafkaStorageException.class", >> 0x7fc92913d950) = -1 ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/common/KafkaException.class", >> 0x7fc92913d950) = -1 ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/utils/Logging$class.class", >> 0x7fc92913de00) = -1 ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/utils/Log4jController$.class", >> 0x7fc92913dc60) = -1 ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/utils/Log4jController.class", >> 0x7fc92913d7c0) = -1 ENOENT (No such file or directory) >> 13342 stat("/var/migration/kafka/utils/Log4jControllerMBean.class", >> 0x7fc92913b4e0) = -1 ENOENT (No such file or directory) >> 13342 >> stat("/var/migration/org/apache/log4j/spi/ThrowableInformation.class", >> 0x7fc92913e860) = -1 ENOENT (No such file or directory) >> >> and after I cd to kafka-0.8.1.1-src/core/build/classes/main/, a lot of >> those ENOENT disappear, but I still get the same "Null Pointer error". >> Grepping "ENOENT" I see: >> >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/Environment.class", >> 0x7fee656ae750) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/Version.class", >> 0x7fee656ae680) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/version/Info.class", >> 0x7fee656ac3a0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/Environment$Entry.class", >> 0x7fee656ae8e0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/ZooKeeper$ZKWatchManager.class", >> 0x7fee656aede0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/ClientCnxn.class", >> 0x7fee656aede0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/proto/RequestHeader.class", >> 0x7fee656ae5f0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/zookeeper/ClientCnxn$1.class", >> 0x7fee656ae9a0) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/META-INF/services/java.nio.channels.spi.SelectorProvider", >> 0x7fee656aed20) = -1 ENOENT (No such file or directory) >> 13257 >> >> stat("/var/migration/migration-tool/kafka-0.8.1.1-src/core/build/classes/main/org/apache/log4j/spi/ThrowableInformation.class", >> 0x7fee656af860) = -1 ENOENT (No such file or directory) >> >> I found, for instance ClientCnxn$1.class in zookeeper-3.3.4.jar >> >> $ jar tf ./core/build/dependant-libs-2.8.0/zookeeper-3.3.4.jar|grep Client >> org/apache/zookeeper/ClientCnxn$1.class >> org/apache/zookeeper/ClientCnxn$2.class >> org/apache/zookeeper/ClientCnxn$AuthData.class >> org/apache/zookeeper/ClientCnxn$EndOfStreamException.class >> org/apache/zookeeper/ClientCnxn$EventThread.class >> org/apache/zookeeper/ClientCnxn$Packet.class >> org/apache/zookeeper/ClientCnxn$SendThread.class >> org/apache/zookeeper/ClientCnxn$SessionExpiredException.class >> org/apache/zookeeper/ClientCnxn$SessionTimeoutException.class >> org/apache/zookeeper/ClientCnxn$WatcherSetEventPair.class >> org/apache/zookeeper/ClientCnxn.class >> org/apache/zookeeper/ClientWatchManager.class >> org/apache/zookeeper/ServerAdminClient.class >> >> And that jarfile is on the classpath, and I see on strace that it's >> finding >> it... Why is it not finding the class, then? >> >> I'm a bit lost... Is there any way I can run it with more verbose? Strace >> is too verbose, "bash -x" is not that much. Something in between? >> >> >> >> >> On Sat, Nov 1, 2014 at 4:53 AM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> >> > The file should be your Scala jar. >> > You should have it in your Kafka lib directory. >> > Do a "jar -t" to make sure it actually contains the Tuple class. >> > >> > Gwen >> > >> > On Fri, Oct 31, 2014 at 8:49 PM, Tomas Nunez <nu...@pythian.com> wrote: >> > >> > > Thanks for pointing the error. I fixed it, but I'm still getting the >> same >> > > error. The content of migrationToolProducer.properties is now: >> > > _____ >> > > >> > > >> > >> metadata.broker.list=kafka_new01:9092,kafka_new02:9092,kafka_new03:9092,kafka_new04:9092,kafka_new05:9092, >> > > group.id=migration.prod >> > > _____ >> > > >> > > And the error still is: >> > > >> > > Exception in thread "main" java.lang.NoClassDefFoundError: >> > > scala/Tuple2$mcLL$sp >> > > at kafka.utils.Utils.stackTrace(Utils.scala) >> > > at >> > kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:268) >> > > Caused by: java.lang.ClassNotFoundException: scala.Tuple2$mcLL$sp >> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> > > at java.security.AccessController.doPrivileged(Native Method) >> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > at >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > ... 2 more >> > > >> > > I have no "Tuple2*" file anywhere in the server. I'm following >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Migrating+from+0.7+to+0.8 >> > > and I can't see there anything about downloading classes, and I don't >> > find >> > > much people with the same problem, which leads me to think that I'm >> doing >> > > something wrong... >> > > >> > > Any idea on what to try? >> > > >> > > Thanks! >> > > >> > > PD: Of course, I'll tell Alex you say hi :) >> > > >> > > On Sat, Nov 1, 2014 at 4:32 AM, Gwen Shapira <gshap...@cloudera.com> >> > > wrote: >> > > >> > > > This is part of Scala, so it should be in the scala-library-...jar >> > > > >> > > > >> > > > On Fri, Oct 31, 2014 at 8:26 PM, Tomas Nunez <nu...@pythian.com> >> > wrote: >> > > > >> > > > > Well... I used strace and I found it was looking for some classes >> in >> > a >> > > > > wrong path. I fixed most of them, but there's one that isn't >> > anywhere, >> > > > > neither the new nor the old kafka servers: >> > > > > >> > > > > $ strace -o logfile -s 1000 -f kafka/bin/kafka-run-class.sh >> > > > > kafka.tools.KafkaMigrationTool --kafka.07.jar >> > > > > migration-tool/kafka-0.7.2.jar --zkclient.01.jar >> > > > > migration-tool/zkclient-0.1.jar >> > > > > --consumer.config=kafka/config/migrationToolConsumer.properties >> > > > > -producer.config=kafka/config/migrationToolProducer.properties >> > > > > --whitelist=.* >> > > > > Exception in thread "main" java.lang.NoClassDefFoundError: >> > > > > scala/Tuple2$mcLL$sp >> > > > > at kafka.utils.Utils.stackTrace(Utils.scala) >> > > > > at >> > > > kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:268) >> > > > > Caused by: java.lang.ClassNotFoundException: scala.Tuple2$mcLL$sp >> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> > > > > at java.security.AccessController.doPrivileged(Native >> Method) >> > > > > at >> java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > > > at >> > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > > > ... 2 more >> > > > > >> > > > > $ grep ENOENT logfile|tail -1 >> > > > > 16484 >> > > > > >> > > > > >> > > > >> > > >> > >> stat("migration-tool/kafka07/core/target/scala_2.8.0/classes/scala/Tuple2$mcLL$sp.class", >> > > > > 0x7f59b27f4950) = -1 ENOENT (No such file or directory) >> > > > > >> > > > > I saw someone had a similar problem in this list, but it's not >> clear >> > > (at >> > > > > least to me) how it was fixed in the end >> > > > > >> > > > > >> > > > >> > > >> > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201302.mbox/%3CCANEBM33rMTZ2UGp2dpaFj8xV=n4=rr9gp3six0sko++nhqv...@mail.gmail.com%3E >> > > > > >> > > > > The JIRA ticket is still untouched >> > > > > https://issues.apache.org/jira/browse/KAFKA-760 >> > > > > >> > > > > May different versions of scala be the cause of this error? I just >> > used >> > > > > "./sbt update" to make sure... >> > > > > >> > > > > If not, where should I find that class file? >> > > > > >> > > > > Thanks again! >> > > > > >> > > > > On Sat, Nov 1, 2014 at 4:05 AM, Tomas Nunez <nu...@pythian.com> >> > wrote: >> > > > > >> > > > > > Hi >> > > > > > >> > > > > > I'm trying to upgrade a 0.7 kafka cluster, but I'm getting an >> > error: >> > > > > > >> > > > > > I created the file migrationToolConsumer.properties with just >> > theinfo >> > > > to >> > > > > > connect to the old cluster: >> > > > > > _____ >> > > > > > >> > zookeeper.connect=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181 >> > > > > > group.id=migration.prod >> > > > > > ______ >> > > > > > >> > > > > > >> > > > > > Then the producer file >> > > > > > _______ >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> zookeeper.connect=zookeeper01_new:2181,zookeeper02_new:2181,zookeeper03_new:2181 >> > > > > > group.id=migration.prod >> > > > > > _______ >> > > > > > >> > > > > > And then I called the migrationtool, and get this error >> > > > > > >> > > > > > ______________ >> > > > > > $ kafka/bin/kafka-run-class.sh kafka.tools.KafkaMigrationTool >> > > > > > --kafka.07.jar migration-tool/kafka-0.7.2.jar --zkclient.01.jar >> > > > > > migration-tool/zkclient-0.1.jar --num.producers 4 >> > > > > > --consumer.config=kafka/config/migrationToolConsumer.properties >> > > > > > -producer.config=kafka/config/migrationToolProducer.properties >> > > > > > --whitelist=.* >> > > > > > Kafka migration tool failed due to: >> > > > > > java.lang.reflect.InvocationTargetException >> > > > > > at >> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > > > > > Method) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> > > > > > at >> > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:526) >> > > > > > at >> > > > > kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:200) >> > > > > > Caused by: java.lang.NoClassDefFoundError: scala/Tuple2$mcLL$sp >> > > > > > at kafka.utils.ZKConfig.<init>(ZkUtils.scala:302) >> > > > > > at >> > > > kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:44) >> > > > > > ... 5 more >> > > > > > Caused by: java.lang.ClassNotFoundException: >> scala.Tuple2$mcLL$sp >> > > > > > at java.lang.ClassLoader.findClass(ClassLoader.java:531) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> kafka.tools.KafkaMigrationTool$ParentLastURLClassLoader$FindClassClassLoader.findClass(KafkaMigrationTool.java:440) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> kafka.tools.KafkaMigrationTool$ParentLastURLClassLoader$ChildURLClassLoader.findClass(KafkaMigrationTool.java:463) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > > > > ... 7 more >> > > > > > >> > > > > > [2014-11-01 02:51:35,124] ERROR Kafka migration tool failed: >> > > > > > (kafka.tools.KafkaMigrationTool) >> > > > > > java.lang.reflect.InvocationTargetException >> > > > > > at >> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > > > > > Method) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> > > > > > at >> > > > > java.lang.reflect.Constructor.newInstance(Constructor.java:526) >> > > > > > at >> > > > > kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:200) >> > > > > > Caused by: java.lang.NoClassDefFoundError: scala/Tuple2$mcLL$sp >> > > > > > at kafka.utils.ZKConfig.<init>(ZkUtils.scala:302) >> > > > > > at >> > > > kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:44) >> > > > > > ... 5 more >> > > > > > Caused by: java.lang.ClassNotFoundException: >> scala.Tuple2$mcLL$sp >> > > > > > at java.lang.ClassLoader.findClass(ClassLoader.java:531) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> kafka.tools.KafkaMigrationTool$ParentLastURLClassLoader$FindClassClassLoader.findClass(KafkaMigrationTool.java:440) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > > > > at >> > > > > > >> > > > > >> > > > >> > > >> > >> kafka.tools.KafkaMigrationTool$ParentLastURLClassLoader$ChildURLClassLoader.findClass(KafkaMigrationTool.java:463) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> > > > > > ... 7 more >> > > > > > __________ >> > > > > > >> > > > > > >> > > > > > The files kafka-0.7.2.jar and zkclient-0.1.jar are the exact >> same >> > > ones >> > > > > > that are being used in the old 0.7 cluster. >> > > > > > >> > > > > > I also tried downloading and building a new 0.7.2 jar file with >> > > scala, >> > > > > but >> > > > > > it had the same error, too. >> > > > > > >> > > > > > What can I be doing wrong? Is there any way to get more output >> to >> > see >> > > > > > where is the error? >> > > > > > >> > > > > > Thanks in advance, >> > > > > > >> > > > > > -- >> > > > > > Tomàs Núñez >> > > > > > Enterprise Infrastructure Consultant >> > > > > > The Pythian Group - Love your data! >> > > > > > >> > > > > > Office (international): +1 613 565 8696 x1501 >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Tomàs Núñez >> > > > > Enterprise Infrastructure Consultant >> > > > > The Pythian Group - Love your data! >> > > > > >> > > > > Office (international): +1 613 565 8696 x1501 >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Tomàs Núñez >> > > Enterprise Infrastructure Consultant >> > > The Pythian Group - Love your data! >> > > >> > > Office (international): +1 613 565 8696 x1501 >> > > >> > > -- >> > > >> > > >> > > -- >> > > >> > > >> > > >> > > >> > >> >> >> >> -- >> Tomàs Núñez >> Enterprise Infrastructure Consultant >> The Pythian Group - Love your data! >> >> Office (international): +1 613 565 8696 x1501 >> >> -- >> >> >> -- >> >> >> >> >