a0x edited a comment on issue #4442:
URL: https://github.com/apache/hudi/issues/4442#issuecomment-1004486507
> I have the same issue when running hudi on emr. This issue seems to have
the same root cause as in this one: #4474 . The solution is to shade and
relocate aws dependencies introduced in hudi-aws:
>
> > For our internal hudi version, we shade aws dependencies, you can add
new relocation and build a new bundle package:
> > For example, to shade aws dependencies in spark, add following codes in
**packaging/hudi-spark-bundle/pom.xml**
> > ```
> > <!-- line 185-->
> > <relocation>
> > <pattern>com.amazonaws.</pattern>
> >
<shadedPattern>${spark.bundle.spark.shade.prefix}com.amazonaws.</shadedPattern>
> > </relocation>
> > ```
>
> @xushiyan should this relocation be added to the official hudi release to
avoid such conflicts?
@kazdy Thank you! This should work.
But shall we shade all aws deps in Spark? I'm worrying about the side
effict, but let me have a try before replying in #4474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
