Re: Sorting on a streaming dataframe

Perhaps your use case fits to Apache Kafka better. More info at: https://kafka.apache.org/documentation/streams/ Everything really comes down to the architecture design and algorithm spec. However, from my experience with Spark, there are many g

Re: spark job scheduling

I think the smallest unit of work is a "Task", and an "Executor" is responsible for getting the work done? Would like to understand more about the scheduling system too. Scheduling strategy like FAIR or FIFO do have significant impact on a Spark cluster architecture design decision. Best, Chayapa

Re: spark job scheduling

k.scheduler.mode as FAIR, and if I submit jobs > without specifying a scheduler pool (which has FAIR scheduling)? would the > jobs still run in FIFO mode with the default pool? > essentially, for us to really set FAIR scheduling, do we have to assign a > FAIR scheduler pool also to the jo