Hi Crystal,
The fair scheduler is only for jobs running concurrently within the same
SparkContext (i.e. within an application), not for separate applications on the
standalone cluster manager. It has no effect there. To run more of those
concurrently, you need to set a cap on how many cores the
Hi
I am trying to switch from FIFO to FAIR with standalone mode.
my environment:
hadoop 1.2.1
spark 0.8.0 using stanalone mode
and i modified the code..
ClusterScheduler.scala -> System.getProperty("spark.scheduler.mode",
"FAIR"))
SchedulerBuilder.scala ->
val DEFAULT_SCHEDULING_MODE
I am loading a csv text file from s3 into spark, filtering and mapping the
records and writing the result to s3.
I have tried several input sizes: 100k rows, 1M rows & 3.5M rows. The former
two finish successfully while the latter (3.5M rows) hangs in some weird
state in which the job stages monit
