Re: Compiling Spark with a local hadoop profile

2015-10-09 Thread Steve Loughran
> On 8 Oct 2015, at 19:31, sbiookag wrote: > > Thanks Ted for reply. > > But this is not what I want. This would tell spark to read hadoop dependency > from maven repository, which is the original version of hadoop. I myslef is > modifying the hadoop code, and wanted to include them inside the

Re: Compiling Spark with a local hadoop profile

2015-10-08 Thread sbiookag
Thanks Ted for reply. But this is not what I want. This would tell spark to read hadoop dependency from maven repository, which is the original version of hadoop. I myslef is modifying the hadoop code, and wanted to include them inside the spark fat jar. "Spark-Class" would run slaves with the fat

Re: Compiling Spark with a local hadoop profile

2015-10-08 Thread Ted Yu
In root pom.xml : 2.2.0 You can override the version of hadoop with command similar to: -Phadoop-2.4 -Dhadoop.version=2.7.0 Cheers On Thu, Oct 8, 2015 at 11:22 AM, sbiookag wrote: > I'm modifying hdfs module inside hadoop, and would like the see the > reflection while i'm running spark on