To help us track planned / finished configuration renames, defaults
changes, and configuration deprecation for the upcoming 1.5.0 release, I
have created https://issues.apache.org/jira/browse/SPARK-9550.
As you make configuration changes or think of configurations that need to
be audited, please u
Thank you for your reply!
Do you mean that currently if i want to use this Tungsten feature, we had to
set sort shuffle manager(spark.shuffle.manager=sort) ,right ? However, I
saw a slide "Deep Dive into Project Tungsten: Bringing Spark Closer to Bare
Metal" published in Spark Summit 2015 and it s
Do we have a data structure that corresponds to buckets in Shuffle ? That is
of we wanted to explore the 'content' of these buckets in shuffle phase, can
we do that ? If yes, how ?
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/What-are-Buckets-referr
There are two usage of buckets used in Spark core.
The first usage is in histogram, used to perform sorting. Basically we
build an approximate histogram of the data in order to decide how to
partition the data in sorting. Each bucket is a range in the histogram.
The 2nd is used in shuffle, where
Hi all ,
I am neebie trying to understand spark internals. There some entity referred
to as 'buckets' at many places in Spark Core code but I am having a hard
time what it is as it is just mentioned in code comments but I didn't come
across any data structure that reffered to it or any class for th
On Sat, Aug 1, 2015 at 1:09 PM Matt Goodman wrote:
> I am considering porting some of this to a more general spark-cloud
> launcher, including google/aliyun/rackspace. It shouldn't be hard at all
> given the current approach for setup/install.
>
FWIW, there are already some tools for launching
