Hi,
Done. See https://github.com/apache/spark/pull/10636
Pozdrawiam,
Jacek
Jacek Laskowski | https://medium.com/@jaceklaskowski/
Mastering Apache Spark
==> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
On Thu, Jan 7, 2016 at 8:10 AM,
The line of code which I highlighted in the screenshot is within the spark
source code. Spark implements sort-based shuffle implementation and the
spilled files are merged using the merge sort.
Here is the link
https://issues.apache.org/jira/secure/attachment/12655884/Sort-basedshuffledesign.pdf
w
The same issue exists in spark-1.6.0
I've opened Jira ticket for that
https://issues.apache.org/jira/browse/SPARK-12655
On Mon, Jan 4, 2016 at 11:30 PM, Alexander Pivovarov
wrote:
> // open spark-shell 1.5.2
> // run
>
> import org.apache.spark.graphx._
>
> val vert = sc.parallelize(List((1L, 1
Here are two interesting issues (with PRs) concerning Java APIs for
Spark 2.x. Details and discussion inside, and comments requested.
https://issues.apache.org/jira/browse/SPARK-3369
This concerns fixing Iterator/Iterable problems in some Java API
methods, to make them consistent with Scala. Thi
If you're filling up the number of open files, odds are there's one code
path that's opening most of these files. If that's the case, these files
will likely be named similarly and easy to pick out if you just sort the
output of "lsof" once you find the group that is clearly the largest, you
can th
Running 'lsof' will let us know the open files but how do we come to know
the root cause behind opening too many files.
Thanks,
Padma CH
On Wed, Jan 6, 2016 at 8:39 AM, Hamel Kothari
wrote:
> The "Too Many Files" part of the exception is just indicative of the fact
> that when that call was mad
