[
https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984771#comment-15984771
]
Steve Loughran commented on SPARK-7481:
---------------------------------------
I think we ended up going in circles on that PR. Sean has actually been very
tolerant of me, however it's been hampered by my full time focus on other
thingsr. I've only been had time to work on the spark PR intermittently and
that's been hard for all: me in the rebase/retest, the one reviewer in having
to catch up again.
Now, anyone who does manage to get that CP right will discover that S3A
absolutely flies with Spark, in partitioning (list file improvements), data
input (set fadvise=true for ORC and Parquet), and for output (set
fast.output=true, play with the pool options). It delivers that performance
because this patch set things up for the integration tests, downstream of this
patch so I and others can be confident that the things actually work, at sped,
at scale. Indeed, many of S3A performance work was actually based on Hive and
Spark workloads:, the data formats & their seek patterns, directory layouts,
file generation. All that's left is the little problem of getting the classpath
right. Oh, and the committer.
For now, for people's enjoyment, here's some videos from Spark Summit East on
the topic
* [Spark and object stores|https://youtu.be/8F2Jqw5_OnI].
* [Robust and Scalable etl over Cloud Storage With
Spark|https://spark-summit.org/east-2017/events/robust-and-scalable-etl-over-cloud-storage-with-spark/]
> Add spark-hadoop-cloud module to pull in object store support
> -------------------------------------------------------------
>
> Key: SPARK-7481
> URL: https://issues.apache.org/jira/browse/SPARK-7481
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 2.1.0
> Reporter: Steve Loughran
>
> To keep the s3n classpath right, to add s3a, swift & azure, the dependencies
> of spark in a 2.6+ profile need to add the relevant object store packages
> (hadoop-aws, hadoop-openstack, hadoop-azure)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
