> On 25 Aug 2017, at 19:49, Allen Wittenauer <a...@effectivemachines.com> wrote:
>
>
>> On Aug 25, 2017, at 10:00 AM, Steve Loughran <ste...@hortonworks.com> wrote:
>>
>> Catching up on this. Looks like I don't have a hadoop-aws profile, which
>> explains a lot, doesn't it.
>
> Yes. This is exactly the type of failure I'd expect.
>
>> How do those profiles get created/copied in?
>
> Maven kludgery.
>
> In a hadoop-tools sub-module pom.xml, you'll find an entry like this or
> similar:
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-dependency-plugin</artifactId>
> <executions>
> <execution>
> <id>deplist</id>
> <phase>compile</phase>
> <goals>
> <goal>list</goal>
> </goals>
> <configuration>
> <!-- build a shellprofile -->
>
> <outputFile>${project.basedir}/target/hadoop-tools-deps/${project.artifactId}.tools-optional.txt</outputFile>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
> The files generated by this entry get read by
> dev-support/bin/dist-tools-hooks-maker. That script is run as part of -Pdist
> in hadoop-dist. The outputFile name determines what kind of support hook it
> makes. (There were a lot of bad decisions made in nomenclature here. I take
> full responsibility for the confusion. But it makes more sense when one
> views the names from the perspective of the code in hadoop-functions.sh)
>
> All/most of this hackery should probably get replaced by something
> smarter in the hadoop-maven-plugin. But for the most part, this does work
> though and makes the end user experience significantly better.
>
>> I know there's an explicit s3guard entry now.
>>
>> hadoop-tools/hadoop-aws/src/main/shellprofile.d/hadoop-s3guard.sh
>>
>> ..do you think the presence of that entry is causing problems (i.e stopping
>> a hadoop-aws profile being created?)
>
> I can confirm that HADOOP-13345 doesn't get a
> shellprofile.d/hadoop-aws.sh created. That's not good. I don't have time
> right now to dig deep, but a few things pop into my head:
>
> * multiple org.apache.maven.plugins definitions in the pom.xml (do all of
> them get executed or just the last one?)
> * dist-tools-hooks-maker may only allowed one of builtin or optional . may
> need to define a 3rd type that does a smart version of both
> * -Pdist may only allow one shellprofile.d dir per module ?
>
> If you want, file a jira and assign it to me. I'll try and dig into it
> next week.
Let me look at it first; I'd want this dealt with before doing the S3Guard
branch merge so there's no regression in trunk
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
