Re: Spark Shuffle - in kubeflow spark operator installation on k8s

icted by kubernetes If anyone has any input on this, pls let me know. thanks! On Sun, Apr 6, 2025 at 8:24 PM karan alang wrote: > Thanks, Megh ! > > I did some research and realized the same - PVC is not a good option for > spark shuffle, primarily for latency issues. &g

Re: Spark Shuffle - in kubeflow spark operator installation on k8s

Thanks, Megh ! I did some research and realized the same - PVC is not a good option for spark shuffle, primarily for latency issues. The same is the case with S3 or MinIO. I've implemented option 2, and am testing this out currently: Storing data in host path is possible regds, Karan

Re: Spark Shuffle - in kubeflow spark operator installation on k8s

;> >>> hello All, >>> >>> I have kubeflow Spark Operator installed on k8s and from what i >>> understand - Spark Shuffle is not officially supported on kubernetes. >>> >>> Looking for feedback from the community on what approach is being taken >&

Re: Spark Shuffle - in kubeflow spark operator installation on k8s

o All, >>> >>> I have kubeflow Spark Operator installed on k8s and from what i >>> understand - Spark Shuffle is not officially supported on kubernetes. >>> >>> Looking for feedback from the community on what approach is being taken >>> to handle th

Re: Spark Shuffle - in kubeflow spark operator installation on k8s

wrote: > >> hello All, >> >> I have kubeflow Spark Operator installed on k8s and from what i >> understand - Spark Shuffle is not officially supported on kubernetes. >> >> Looking for feedback from the community on what approach is being taken >> t

Re: Spark Shuffle - in kubeflow spark operator installation on k8s

hello all - checking to see if anyone has any input on this thanks! On Tue, Mar 25, 2025 at 12:22 PM karan alang wrote: > hello All, > > I have kubeflow Spark Operator installed on k8s and from what i understand > - Spark Shuffle is not officially supported on kubernetes. >

Spark Shuffle - in kubeflow spark operator installation on k8s

hello All, I have kubeflow Spark Operator installed on k8s and from what i understand - Spark Shuffle is not officially supported on kubernetes. Looking for feedback from the community on what approach is being taken to handle this issue - especially since dynamicAllocation cannot be enabled

Re: External Spark shuffle service for k8s

I think this answers your question about what to do if you need more space on nodes. https://spark.apache.org/docs/latest/running-on-kubernetes.html#local-storage Local Storage Spark supports using volumes to spill d

Re: External Spark shuffle service for k8s

" In the end for my usecase I started using pvcs and pvc aware scheduling along with decommissioning. So far performance is good with this choice." How did you do this? tor. 11. apr. 2024 kl. 04:13 skrev Arun Ravi : > Hi Everyone, > > I had to explored IBM's and AWS's S3 shuffle plugins (some ti

Re: External Spark shuffle service for k8s

Hi Everyone, I had to explored IBM's and AWS's S3 shuffle plugins (some time back), I had also explored AWS FSX lustre in few of my production jobs which has ~20TB of shuffle operations with 200-300 executors. What I have observed is S3 and fax behaviour was fine during the write phase, however I

Re: External Spark shuffle service for k8s

Hi, First thanks everyone for their contributions I was going to reply to @Enrico Minack but noticed additional info. As I understand for example, Apache Uniffle is an incubating project aimed at providing a pluggable shuffle service for Spark. So basically, all these "external shuffle service

Re: External Spark shuffle service for k8s

I see that both Uniffle and Celebron support S3/HDFS backends which is great. In the case someone is using S3/HDFS, I wonder what would be the advantages of using Celebron or Uniffle vs IBM shuffle service plugin or Cloud Shuffle Storage Plugin from AWS

Re: External Spark shuffle service for k8s

Apache Uniffle (incubating) may be another solution. You can see https://github.com/apache/incubator-uniffle https://uniffle.apache.org/blog/2023/07/21/Uniffle%20-%20New%20chapter%20for%20the%20shuffle%20in%20the%20cloud%20native%20era Mich Talebzadeh 于2024年4月8日周一 07:15写道： > Splendid > > The con

Re: External Spark shuffle service for k8s

There is Apache incubator project Uniffle: https://github.com/apache/incubator-uniffle It stores shuffle data on remote servers in memory, on local disk and HDFS. Cheers, Enrico Am 06.04.24 um 15:41 schrieb Mich Talebzadeh: I have seen some older references for shuffle service for k8s, althou

Re: External Spark shuffle service for k8s

Thanks Cheng for the heads up. I will have a look. Cheers Mich Talebzadeh, Technologist | Solutions Architect | Data Engineer | Generative AI London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebz

Re: External Spark shuffle service for k8s

There is an IBM shuffle service plugin that supports S3 https://github.com/IBM/spark-s3-shuffle Though I would think a feature like this could be a part of the main Spark repo. Trino already has out-of-box support for s3 exchange (shuffle) and it's very useful. Vakaris On Sun, Apr 7, 2024 at 12:

Re: External Spark shuffle service for k8s

Instead of External Shuffle Shufle, Apache Celeborn might be a good option as a Remote Shuffle Service for Spark on K8s. There are some useful resources you might be interested in. [1] https://celeborn.apache.org/ [2] https://www.youtube.com/watch?v=s5xOtG6Venw [3] https://github.com/aws-samples

Re: External Spark shuffle service for k8s

Splendid The configurations below can be used with k8s deployments of Spark. Spark applications running on k8s can utilize these configurations to seamlessly access data stored in Google Cloud Storage (GCS) and Amazon S3. For Google GCS we may have spark_config_gcs = { "spark.kubernetes.auth

Re: External Spark shuffle service for k8s

Thanks for your suggestion that I take it as a workaround. Whilst this workaround can potentially address storage allocation issues, I was more interested in exploring solutions that offer a more seamless integration with large distributed file systems like HDFS, GCS, or S3. This would ensure bette

Re: External Spark shuffle service for k8s

You can make a PVC on K8S call it 300GB make a folder in yours dockerfile WORKDIR /opt/spark/work-dir RUN chmod g+w /opt/spark/work-dir start spark with adding this .config("spark.kubernetes.driver.volumes.persistentVolumeClaim.300gb.options.claimName", "300gb") \ .config("spark.kubernetes.driv

External Spark shuffle service for k8s

I have seen some older references for shuffle service for k8s, although it is not clear they are talking about a generic shuffle service for k8s. Anyhow with the advent of genai and the need to allow for a larger volume of data, I was wondering if there has been any more work on this matter. Speci

Re: Spark shuffle and inevitability of writing to Disk

2| 1| +---+---+ Now I have attached the UI plot as well. The section on "Aggregated Metric by Executor" shows columns Spill(Memory) and Spill(Disk) highlighted in yellow circle. My deduction is that Spark will try to use memory for shuffle if it can but will revert

Spark shuffle and inevitability of writing to Disk

Hi, On the issue of Spark shuffle it is accepted that shuffle *often involves* the following if not all below: - Disk I/O - Data serialization and deserialization - Network I/O Excluding external shuffle service and without relying on the configuration options provided by spark for

Re: understanding spark shuffle file re-use better

Increase or Decrease the number of data partitions: Since a data partition represents the quantum of data to be processed together by a single Spark Task, there could be situations: (a) Where existing number of data partitions are not sufficient enough in order to maximize the usage of available r

Re: understanding spark shuffle file re-use better

A much better one-liner (easier to understand the UI because it will be 1 simple job with 2 stages): ``` spark.read.text("README.md").repartition(2).take(1) ``` Attila Zsolt Piros wrote > No, it won't be reused. > You should reuse the dateframe for reusing the shuffle blocks (and cached > data).

Re: understanding spark shuffle file re-use better

No, it won't be reused. You should reuse the dateframe for reusing the shuffle blocks (and cached data). I know this because the two actions will lead to building a two separate DAGs, but I will show you a way how you could check this on your own (with a small simple spark application). For this

Re: understanding spark shuffle file re-use better

Hi, An interesting question that I must admit I'm not sure how to answer myself actually :) Off the top of my head, I'd **guess** unless you cache the first query these two queries would share nothing. With caching, there's a phase in query execution when a canonicalized version of a query is use

understanding spark shuffle file re-use better

is shuffle file re-use based on identity or equality of the dataframe? for example if run the exact same code twice to load data and do transforms (joins, aggregations, etc.) but without re-using any actual dataframes, will i still see skipped stages thanks to shuffle file re-use? thanks! koert

Re: [apache-spark]-spark-shuffle

How a Spark job reads datasources depends on the underlying source system,the job configuration about number of executors and cores per executor. https://spark.apache.org/docs/latest/rdd-programming-guide.html#external-datasets About Shuffle operations. https://spark.apache.org/docs/latest/rdd-p

[apache-spark]-spark-shuffle

Hi, I am trying to thoroughly understand below concepts in spark. 1. A job is reading 2 files and performing a cartesian join. 2. Sizes of input are 55.7 mb and 67.1 mb 3. after reading input file, spark did shuffle, for both the inputs shuffle was in KB. I want to understand why this size is not

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

If not resilient at spark level, can't you just relaunch you job with your orchestration tool ? Le 21 déc. 2017 09:34, "Georg Heiler" a écrit : > Die you try to use the yarn Shuffle Service? > chopinxb schrieb am Do. 21. Dez. 2017 um 04:43: > >> In my practice of spark application(almost Spark-

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

Die you try to use the yarn Shuffle Service? chopinxb schrieb am Do. 21. Dez. 2017 um 04:43: > In my practice of spark application(almost Spark-SQL) , when there is a > complete node failure in my cluster, jobs which have shuffle blocks on the > node will completely fail after 4 task retries. It

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

In my practice of spark application(almost Spark-SQL) , when there is a complete node failure in my cluster, jobs which have shuffle blocks on the node will completely fail after 4 task retries. It seems that data lineage didn't work. What' more, our applications use multiple SQL statements for da

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

Probability of a complete node failure is low. I would rely on data lineage and accept the reprocessing overhead. Another option would be to Write on distributed FS but it will drastically reduce all your jobs speed Le 20 déc. 2017 11:23, "chopinxb" a écrit : > Yes，shuffle service was already st

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

Yes，shuffle service was already started in each NodeManager. What i mean about node fails is the machine is down，all the service include nodemanager process in this machine is down. So in this case, shuffle service is no longer helpfull -- Sent from: http://apache-spark-user-list.1001560.n3.nab

Re: Can spark shuffle leverage Alluxio to abtain higher stability？

In your case you need to externalize the shuffle files to a component outside of your spark cluster to make them persist after spark workers death. https://spark.apache.org/docs/latest/running-on-yarn.html#configuring-the-external-shuffle-service 2017-12-20 10:46 GMT+01:00 chopinxb : > In my use

Can spark shuffle leverage Alluxio to abtain higher stability？

In my use case, i run spark on yarn-client mode with dynamicAllocation enabled, When a node shutting down abnormally, my spark application will fails because of task fail to fetch shuffle blocks from that node 4 times. Why spark do not leverage Alluxio（distributed in-memory filesystem) to write sh

Re: Spark Shuffle Encryption

http://spark.apache.org/docs/latest/configuration.html#shuffle-behavior All the options you need to know are there. On Fri, May 12, 2017 at 9:11 AM, Shashi Vishwakarma wrote: > Hi > > I was doing research on encrypting spark shuffle data and found that Spark > 2.1 has got

Spark Shuffle Encryption

Hi I was doing research on encrypting spark shuffle data and found that Spark 2.1 has got that feature. https://issues.apache.org/jira/browse/SPARK-5682 Does anyone has more documentation around it ? How do I aim to use this feature in real production environment keeping mind that I need to

Re: Spark shuffle files

When the RDD using them goes out of scope. On Mon, Mar 27, 2017 at 3:13 PM, Ashwin Sai Shankar wrote: > Thanks Mark! follow up question, do you know when shuffle files are > usually un-referenced? > > On Mon, Mar 27, 2017 at 2:35 PM, Mark Hamstra > wrote: > >> Shuffle files are cleaned when the

Re: Spark shuffle files

Thanks Mark! follow up question, do you know when shuffle files are usually un-referenced? On Mon, Mar 27, 2017 at 2:35 PM, Mark Hamstra wrote: > Shuffle files are cleaned when they are no longer referenced. See > https://github.com/apache/spark/blob/master/core/src/ > main/scala/org/apache/spar

Re: Spark shuffle files

Shuffle files are cleaned when they are no longer referenced. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ContextCleaner.scala On Mon, Mar 27, 2017 at 12:38 PM, Ashwin Sai Shankar < ashan...@netflix.com.invalid> wrote: > Hi! > > In spark on yarn, when are

Spark shuffle files

Hi! In spark on yarn, when are shuffle files on local disk removed? (Is it when the app completes or once all the shuffle files are fetched or end of the stage?) Thanks, Ashwin

Re: Spark shuffle: FileNotFound exception

Swapnil, What do you think might be the size of the file that’s not found? For spark version below 2.0.0 there might be issues with blocks of size 2g. Is the file actually on a file system? I’d try to increase default parallelism to make sure partitions got smaller. Hope, this helps. > On 04

Spark shuffle: FileNotFound exception

Hello All I am facing FileNotFoundException for shuffle index file when running job with large data. Same job runs fine with smaller datasets. These our my cluster specifications - No of nodes - 19 Total cores - 380 Memory per executor - 32G Spark 1.6 mapr version spark.shuffle.service.enabled

Re: Spark Shuffle Issue

Hi, I was able to resolve the issue with increasing the timeout and reducing the number of executors and increasing number of cores per executor. The issue is resolved but I am still not sure why reducing the number of executors and increasing number of cores per executor fixed issues related to

Spark Shuffle Issue

Hi, I am upgrading my jobs to Spark 1.6 and am running into shuffle issues. I have tried all options and now am falling back to legacy memory model but still running into same issue. I have set spark.shuffle.blockTransferService to nio. 16/10/12 06:00:10 INFO MapOutputTrackerMaster: Size of outp

Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

d.com <mailto:tony@tendcloud.com> > > From: Sun Rui <mailto:sunrise_...@163.com> > Date: 2016-08-24 22:17 > To: Saisai Shao <mailto:sai.sai.s...@gmail.com> > CC: tony@tendcloud.com <mailto:tony@tendcloud.com>; user > <mailto:user@spark.apa

Re: Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

loud.com > > > *From:* Sun Rui > *Date:* 2016-08-24 22:17 > *To:* Saisai Shao > *CC:* tony....@tendcloud.com; user > *Subject:* Re: Can we redirect Spark shuffle spill data to HDFS or > Alluxio? > Yes, I also tried FUSE before, it is not stable and I don’t recommend it &

Re: Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

@tendcloud.com From: Sun Rui Date: 2016-08-24 22:17 To: Saisai Shao CC: tony@tendcloud.com; user Subject: Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio? Yes, I also tried FUSE before, it is not stable and I don’t recommend it On Aug 24, 2016, at 22:15, Saisai Shao wrote

Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

> >> On Aug 24, 2016, at 21:02, Saisai Shao > <mailto:sai.sai.s...@gmail.com>> wrote: >> >> Spark Shuffle uses Java File related API to create local dirs and R/W data, >> so it can only be worked with OS supported FS. It doesn't leverage Hadoop >> F

Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

ead of network I/O and replica > of HDFS files. > > On Aug 24, 2016, at 21:02, Saisai Shao wrote: > > Spark Shuffle uses Java File related API to create local dirs and R/W > data, so it can only be worked with OS supported FS. It doesn't leverage > Hadoop FileSystem API, so wr

Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

For HDFS, maybe you can try mount HDFS as NFS. But not sure about the stability, and also there is additional overhead of network I/O and replica of HDFS files. > On Aug 24, 2016, at 21:02, Saisai Shao wrote: > > Spark Shuffle uses Java File related API to create local dirs and R/W da

Re: Can we redirect Spark shuffle spill data to HDFS or Alluxio?

Spark Shuffle uses Java File related API to create local dirs and R/W data, so it can only be worked with OS supported FS. It doesn't leverage Hadoop FileSystem API, so writing to Hadoop compatible FS is not worked. Also it is not suitable to write temporary shuffle data into distributed FS,

Can we redirect Spark shuffle spill data to HDFS or Alluxio?

Hi, All, When we run Spark on very large data, spark will do shuffle and the shuffle data will write to local disk. Because we have limited capacity at local disk, the shuffled data will occupied all of the local disk and then will be failed. So is there a way we can write the shuffle spill dat

Re: Re: how to tuning spark shuffle

this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-tuning-spark-shuffle-tp27350.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >

Re: how to tuning spark shuffle

this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/how-to-tuning-spark-shuffle-tp27350.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >

how to tuning spark shuffle

.1001560.n3.nabble.com/how-to-tuning-spark-shuffle-tp27350.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: In-Memory Only Spark Shuffle

en-sort shuffle manager; its optimized shuffles > will > continue to spill to disk when necessary.” > > If this is impossible via configuration changes what code changes would be > needed to accomplish this? > > > > -- > View this message in context: > http://apac

In-Memory Only Spark Shuffle

uffle manager; its optimized shuffles will continue to spill to disk when necessary.” If this is impossible via configuration changes what code changes would be needed to accomplish this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/In-Memory-Only-

Re: spark shuffle service on yarn

If you use any shuffle service before 2.0 it should be compatible with all previous releases. The 2.0 version has currently an incompatibility that we should probably patch before releasing 2.0, to support this kind of use case (among others). On Fri, Mar 18, 2016 at 7:25 PM, Koert Kuipers wrote

Re: spark shuffle service on yarn

On 19 Mar 2016, at 02:25, Koert Kuipers mailto:ko...@tresata.com>> wrote: spark on yarn is nice because i can bring my own spark. i am worried that the shuffle service forces me to use some "sanctioned" spark version that is officially "installed" on the cluster. so... can i safely install th

spark shuffle service on yarn

spark on yarn is nice because i can bring my own spark. i am worried that the shuffle service forces me to use some "sanctioned" spark version that is officially "installed" on the cluster. so... can i safely install the spark 1.3 shuffle service on yarn and use it with other 1.x versions of spark

Spark Partitioner vs Spark Shuffle Manager

Hi All, What is the difference between Spark Partitioner and Spark Shuffle Manager. Spark Partitioner is by default Hash partitioner and Spark shuffle manager is sort based, others are Hash, Tunsten Sort. Thanks, Prabhu Joseph

Questions about Spark Shuffle and Heap

might be used instead during shuffles. Do these two explanation conflict each other? In a related question, with Tungsten enabled, when does Spark use off-heap memory? Thanks, Jianneng -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark

Re: spark shuffle

And does groupByKey will keep all values of pairrdd in an iterable list in inmemory of reducer. Which will lead to outofmemory if values of a key are beyond memory of that node . 1.Is there a way to spill that to disk ? 2.If not is there a feasibility of partitioning pairdd using custom partitione

spark shuffle

Hi I have few doubts 1.does rdd.saveasNewAPIHadoopFile(outputdir,keyclass,valueclass,ouputformat class)-> shuffles data or it will always create same no of files in output dir as number of partitions in rdd. 2. How to use multiple outputs in saveasNewAPIHadoopFile to have file name generated fro

What would happen when reduce memory is not enough on spark shuffle read stage?

-would-happen-when-reduce-memory-is-not-enough-on-spark-shuffle-read-stage-tp25330.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For

Question about Spark shuffle read size

Hi all, When I run WordCount using Spark, I find that when I set "spark.default.parallelism" to different numbers, the Shuffle Write size and Shuffle Read size will change as well (I read these data from history server's web UI). Is it because the shuffle write size also include some metadata size

RE: Spark shuffle service does not work in stand alone

Thanks, I missed that one. From: Marcelo Vanzin [mailto:van...@cloudera.com] Sent: Tuesday, October 13, 2015 2:36 PM To: Ellafi, Saif A. Cc: user@spark.apache.org Subject: Re: Spark shuffle service does not work in stand alone You have to manually start the shuffle service if you're not ru

Re: Spark shuffle service does not work in stand alone

You have to manually start the shuffle service if you're not running YARN. See the "sbin/start-shuffle-service.sh" script. On Tue, Oct 13, 2015 at 10:29 AM, wrote: > I believe the confusion here is self-answered. > > The thing is that in the documentation, the spark shu

RE: Spark shuffle service does not work in stand alone

I believe the confusion here is self-answered. The thing is that in the documentation, the spark shuffle service runs only under YARN, while here we are speaking about a stand alone cluster. The proper question is, how to launch a shuffle service for stand alone? Saif From: saif.a.ell

RE: Spark shuffle service does not work in stand alone

[mailto:van...@cloudera.com] Sent: Tuesday, October 13, 2015 1:13 PM To: Ellafi, Saif A. Cc: user@spark.apache.org Subject: Re: Spark shuffle service does not work in stand alone It would probably be more helpful if you looked for the executor error and posted it. The screenshot you posted is the driver

Re: Spark shuffle service does not work in stand alone

It would probably be more helpful if you looked for the executor error and posted it. The screenshot you posted is the driver exception caused by the task failure, which is not terribly useful. On Tue, Oct 13, 2015 at 7:23 AM, wrote: > Has anyone tried shuffle service in Stand Alone cluster mode

Re: Spark shuffle service does not work in stand alone

Hi, AFAIK, the shuffle service makes sense only to delegate the shuffle to mapreduce (as mapreduce shuffle is most of the time faster than the spark shuffle). As you run in standalone mode, shuffle service will use the spark shuffle. Not 100% thought. Regards JB On 10/13/2015 04:23 PM

Spark shuffle service does not work in stand alone

Has anyone tried shuffle service in Stand Alone cluster mode? I want to enable it for d.a. but my jobs never start when I submit them. This happens with all my jobs. 15/10/13 08:29:45 INFO DAGScheduler: Job 0 failed: json at DataLoader.scala:86, took 16.318615 s Exception in thread "main" org.

Re: How to avoid Spark shuffle spill memory?

> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-Spark-shuffle-spill-memory-tp24960.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail

How to avoid Spark shuffle spill memory?

executor to timeout and then getting lost eventually. Please guide. Thanks in advance. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-Spark-shuffle-spill-memory-tp24960.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Debugging too many files open exception issue in Spark shuffle

p 2015 15:53:54 -0700 >> Subject: Debugging too many files open exception issue in Spark shuffle >> From: dbt...@dbtsai.com >> To: user@spark.apache.org > >> >> Hi, >> >> Recently, we ran into this notorious exception while doing large >> shuffle in mesos a

RE: Debugging too many files open exception issue in Spark shuffle

That is interesting. I don't have any Mesos experience, but just want to know the reason why it does so. Yong > Date: Wed, 23 Sep 2015 15:53:54 -0700 > Subject: Debugging too many files open exception issue in Spark shuffle > From: dbt...@dbtsai.com > To: user@spark.a

Debugging too many files open exception issue in Spark shuffle

Hi, Recently, we ran into this notorious exception while doing large shuffle in mesos at Netflix. We ensure that `ulimit -n` is a very large number, but still have the issue. It turns out that mesos overrides the `ulimit -n` to a small number causing the problem. It's very non-trivial to debug (a

Re: Limit Spark Shuffle Disk Usage

Thanks Himanshu and RahulKumar! The databricks forum post was extremely useful. It is great to see an article that clearly details how and when shuffles are cleaned up. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage

Re: Limit Spark Shuffle Disk Usage

6 2 : set 'spark.shuffle.memoryFraction' to 0.4 which is by default 0.2 this should make a significant difference in disk use of shuffle. Thank you - Himanshu Mehra -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279p23334.html Sent f

Re: Limit Spark Shuffle Disk Usage

igmoid (SigmoidAnalytics), India -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279p23323.html Sent from the Apache Spark User List mailing list archive at Nabb

Re: Limit Spark Shuffle Disk Usage

is still running, but I'm concerned that they are left there for a good > reason. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle

Limit Spark Shuffle Disk Usage

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e