The more I think about it the problem is not about /tmp, its more about the workers not having enough memory. Blocks of received data could be falling out of memory before it is getting processed. BTW, what is the storage level that you are using for your input stream? If you are using MEMORY_ONLY, then try MEMORY_AND_DISK. That is safer because it ensure that if received data falls out of memory it will be at least saved to disk.
TD On Thu, Mar 27, 2014 at 2:29 PM, Scott Clasen <scott.cla...@gmail.com>wrote: > Heh sorry that wasnt a clear question, I know 'how' to set it but dont know > what value to use in a mesos cluster, since the processes are running in > lxc > containers they wont be sharing a filesystem (or machine for that matter) > > I cant use an s3n:// url for local dir can I? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3373.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
