Re: Spark 3: Resource Discovery

2021-07-17 Thread Sean Owen
At the moment this is really about discovering GPUs, so that the scheduler can schedule tasks that need to allocate whole GPUs. On Sat, Jul 17, 2021 at 5:14 PM ayan guha wrote: > Hi > > As I was going through Spark 3 config params, I noticed following group of > params. I could not understand wh

Spark 3: Resource Discovery

2021-07-17 Thread ayan guha
Hi As I was going through Spark 3 config params, I noticed following group of params. I could not understand what are they for. Can anyone please point me in the right direction? spark.driver.resource.{resourceName}.amount 0 Amount of a particular resource type to use on the driver. If this is us