If you work for a certain hardware vendor that builds expensive, high performance nodes, and want to use Spark to demonstrate the performance gains of your new great systems, you will of course totally disagree.
Anyway - I offered you a simple solution to work around the low hanging fruits. Feel free to totally disagree and reject that. Yes you might see problems with kernel being unable to manage the buffer pool as well as Spark itself can, but you also might not because most of the software stack (not just Spark, but software in general) are in general inefficient and far away from what the hardware can do at its limit, so having minor, or sometimes even major imperfections somewhere in the stack isn't necessary a problem. For example, you will find that the network software stack in Spark (or majority of the open source projects you will find) actually won't be able to saturate a 10G network in practical jobs, let alone 40G. Decryption, deserialization, or data processing themselves can be expensive, to the point that it doesn't really matter how high your disk throughput or network throughput is. While I think 40G network is coming, they are far away from ubiquity. Does that mean we shouldn't care? No. But it takes time and resources to address them, and in most cases they are not actually the bottleneck. It is not as simple as putting the data in memory, because we'd need to build a bunch of machinery to share that limited memory with the execution part, which have been by far the largest bottleneck. So what does it take to improve this? First and foremost, we would need to substantially speed up the execution engine itself. We are making great progress in Spark 2.0. For a lot of the common SQL-like operations, Spark 2.0 can be pretty fast (e.g. filtering 1 billion records a second, or joining 100 million records a second; using a single core). However, I still don't think it matters much disk vs memory for temporary shuffle files in a moderately sized cluster with SSDs, until we rewrite the network stack to be able to sustainably saturate 10G links. Spark was able to do that two years ago when I first implemented the current network module, but I'm sure after two years of feature development, bug fixes, and security improvement, the network module can no longer do that. Why haven't we fixed it yet? Because most workloads don't shuffle enormous amount of data and when they do, they are not bounded by slower network stack (either software or hardware). One environment this issue would matter a lot is on clusters with a small number of nodes. In the most extreme case with only a single node, the current way we do shuffle in Spark is one to two orders of magnitude slower than some simple in-memory data partitioning algorithm (e.g. radix sort). Addressing that can speed up certain Spark workloads (large joins, large aggregations) quite a bit. On Fri, Apr 1, 2016 at 2:22 PM, Reynold Xin <r...@databricks.com> wrote: > Sure - feel free to totally disagree. > > > On Fri, Apr 1, 2016 at 2:10 PM, Michael Slavitch <slavi...@gmail.com> > wrote: > >> I totally disagree that it’s not a problem. >> >> - Network fetch throughput on 40G Ethernet exceeds the throughput of NVME >> drives. >> - What Spark is depending on is Linux’s IO cache as an effective buffer >> pool This is fine for small jobs but not for jobs with datasets in the >> TB/node range. >> - On larger jobs flushing the cache causes Linux to block. >> - On a modern 56-hyperthread 2-socket host the latency caused by multiple >> executors writing out to disk increases greatly. >> >> I thought the whole point of Spark was in-memory computing? It’s in fact >> in-memory for some things but use spark.local.dir as a buffer pool of >> others. >> >> *Hence, the performance of Spark is gated by the performance of >> spark.local.dir, even on large memory systems.* >> >> "Currently it is not possible to not write shuffle files to disk.” >> >> What changes >would< make it possible? >> >> The only one that seems possible is to clone the shuffle service and make >> it in-memory. >> >> >> >> >> >> On Apr 1, 2016, at 4:57 PM, Reynold Xin <r...@databricks.com> wrote: >> >> spark.shuffle.spill actually has nothing to do with whether we write >> shuffle files to disk. Currently it is not possible to not write shuffle >> files to disk, and typically it is not a problem because the network fetch >> throughput is lower than what disks can sustain. In most cases, especially >> with SSDs, there is little difference between putting all of those in >> memory and on disk. >> >> However, it is becoming more common to run Spark on a few number of beefy >> nodes (e.g. 2 nodes each with 1TB of RAM). We do want to look into >> improving performance for those. Meantime, you can setup local ramdisks on >> each node for shuffle writes. >> >> >> >> On Fri, Apr 1, 2016 at 11:32 AM, Michael Slavitch <slavi...@gmail.com> >> wrote: >> >>> Hello; >>> >>> I’m working on spark with very large memory systems (2TB+) and notice >>> that Spark spills to disk in shuffle. Is there a way to force spark to >>> stay in memory when doing shuffle operations? The goal is to keep the >>> shuffle data either in the heap or in off-heap memory (in 1.6.x) and never >>> touch the IO subsystem. I am willing to have the job fail if it runs out >>> of RAM. >>> >>> spark.shuffle.spill true is deprecated in 1.6 and does not work in >>> Tungsten sort in 1.5.x >>> >>> "WARN UnsafeShuffleManager: spark.shuffle.spill was set to false, but >>> this is ignored by the tungsten-sort shuffle manager; its optimized >>> shuffles will continue to spill to disk when necessary.” >>> >>> If this is impossible via configuration changes what code changes would >>> be needed to accomplish this? >>> >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >
