Re: Including dependencies in a Storm topology jar

[Patrick May]

I’m not a fan of Maven.

I did find a solution.  Apparently nested jars are not supported by Storm.  The 
fix is to unpack the jar files and include the class files in the topology jar. 
This took two changes to my Ant build.xml file. First, put the dependencies in 
a staging directory and build a single jar from them:

  <target name="staging">
    <copy todir="${build.staging}">
      <fileset dir="${kafka.lib}">
        <include name="kafka-clients-0.8.2.1.jar"/>
        <include name="kafka_2.11-0.8.2.1.jar"/>
        <include name="log4j-1.2.16.jar"/>
        <include name="metrics-core-2.2.0.jar"/>
        <include name="scala-library-2.11.5.jar"/>
      </fileset>

      <fileset dir="${storm.kafka.lib}">
        <include name="storm-kafka-0.9.5.jar"/>
      </fileset>

      <fileset dir="${zookeeper.lib}">
        <include name="zookeeper-3.4.6.jar"/>
      </fileset>
    </copy>

    <jar jarfile="${staging.jar}">
      <zipgroupfileset dir="${build.staging}" includes="**/*.jar"/>
    </jar>
  </target>

Then pull those .class files into the topology jar, excluding the manifests:

<jar basedir="${build.root}"
     jarfile="${topology.jar}">
  <manifest>
    <attribute name="Main-Class" value="${topology.fullname}"/>
  </manifest>

  <zipfileset src="${staging.jar}" excludes="META-INF/**/*"/>
</jar>

This jar successfully deploys to Storm.

I’ve put a working example up on Github:  
https://github.com/patrickmay/kafka-storm  (the README.md needs some work).

Thanks,

Patrick


From: Javier Gonzalez
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>"
Date: Monday, September 14, 2015 at 5:37 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>"
Subject: Re: Including dependencies in a Storm topology jar


We use the shaded jar maven plugin. Just make sure that you mark storm as scope 
provided (so that you don't get a duplicate storm jar error) and exclude any 
RSA/DSA/SF signature files from the manifest folder (so that you don't get 
failed signature check errors).

Regards,
Javier

On Sep 14, 2015 1:32 PM, "Patrick May" 
<patrick....@ignitionone.com<mailto:patrick....@ignitionone.com>> wrote:

I'm trying to include some jars required by my topology in the jar that I 
deploy to Storm. When I create the jar file using Ant with the 
manifestclasspath task, I get a manifest that looks like this:

Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: ../build/lib/kafka-clients-0.8.2.1.jar ../build/lib/kafka_
 2.11-0.8.2.1.jar ../build/lib/log4j-1.2.16.jar ../build/lib/metrics-c
 ore-2.2.0.jar ../build/lib/scala-library-2.11.5.jar ../build/lib/stor
 m-kafka-0.9.5.jar ../build/lib/zookeeper-3.4.6.jar

The paths in the Class-Path entry look wrong. The topology deploys 
successfully, but when I send data to Kafka, I get a ClassNotFoundException: 
storm.kafka.KafkaSpout in the Storm logs.

I modified the manifest to look like this:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: . lib/kafka-clients-0.8.2.1.jar lib/kafka_2.11-0.8.2.1.jar
 lib/log4j-1.2.16.jar lib/metrics-core-2.2.0.jar lib/scala-library-2.1
 1.5.jar lib/storm-kafka-0.9.5.jar lib/zookeeper-3.4.6.jar

This also deploys successfully but gives the same error when I try to pass 
messages.

How can I create a jar file that contains these dependencies and runs properly 
on Storm? I don't want to have to add the jars to the Storm lib directory, but 
that's the only configuration I've found that works.