hi all, I have a few questions on spark job scheduling.
1. As I understand, the smallest unit of work an executor can perform. In the 'fair' scheduler mode, let's say a job is submitted to the spark ctx which has a considerable amount of work to do in a task. While such a 'big' task is running, can we still submit another smaller job (from a separate thread) and get it done? or does that smaller job has to wait till the bigger task finishes and the resources are freed from the executor? 2. When a job is submitted without setting a scheduler pool, the default scheduler pool is assigned to it, which employs FIFO scheduling. but what happens when we have the spark.scheduler.mode as FAIR, and if I submit jobs without specifying a scheduler pool (which has FAIR scheduling)? would the jobs still run in FIFO mode with the default pool? essentially, for us to really set FAIR scheduling, do we have to assign a FAIR scheduler pool? best -- Niranda @n1r44 <https://twitter.com/N1R44> +94-71-554-8430 https://pythagoreanscript.wordpress.com/
