I faced exact same issue. The way i solved it was 1. Copy entire project. 2. Delete all the source, have only the dependencies in pom.xml. This will create, fat jar, without source but deps only. 3. In original project keep it as is, now build it. this will create a JAR (no deps, by default)
Now add your deps jar to spark, with --jar This requires two projects, but thats better than earlier. On Mon, Apr 13, 2015 at 4:45 PM, Michael Weir <michael.weir....@gmail.com> wrote: > My app works fine with the single, "uber" jar file containing my app and > all its dependencies. However, it takes about 20 minutes to copy the 65MB > jar file up to the node on the cluster, so my "code, compile, test" cycle > has become a "core, compile, coooooopppppppyyyy, test" cycle. > > I'd like to have a single dependencies jar file on the node, and use a > separate small jar for my app (which takes around 10 seconds to copy to the > node). > > I've tried using "--jars deps.jar", but that copies the deps.jar to the > app-* folder but not the driver-* folder, so I get classNotFound errors on > the driver. Various other combinations of flags, etc. have produced a fair > bit of frustration but no progress. > > Any help with this would be greatly appreciated, as this problem is > significantly stretching the length of my work day! > > Thanks. > -- Deepak
