[
https://issues.apache.org/jira/browse/SPARK-30451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939692#comment-17939692
]
Jayadeep Jayaraman edited comment on SPARK-30451 at 3/31/25 1:23 PM:
---------------------------------------------------------------------
Hi [~tgraves] - I would like to work on this request. As per the description my
understanding is
{code:java}
class ExecutorResourceRequests() {code}
is a convenience method for the class
{code:java}
class ExecutorResourceRequest(){code}
Class `ExecutorResourceRequests() ` has multiple methods to add different
resources
{code:java}
def offHeapMemory(amount: String): this.type
def memoryOverhead(amount: String): this.type
def memory(amount: String): this.type{code}
and they are stored in a mutable map
{code:java}
new ConcurrentHashMap[String, ExecutorResourceRequest](){code}
The goal of this PR is to offer an API that allows users to build a Request
profile once as shown below
{code:java}
val executorReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder()
.memory("1g") .cores(4) .resource("gpu", 2) .build()
{code}
and then offer ways to add or remove specific resources as shown below
{code:java}
val anotherReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder()
.offHeapMemory("512m") .add(new ExecutorResourceRequest("fpga", 1))
.remove("memory") // This would have no effect as 'memory' wasn't added in this
builder .build()
{code}
Kindly, let me know if my understanding is correct and I would like to work on
this task
was (Author: jjayadeep):
Hi [~tgraves] - I would like to work on this request. As per the description my
understanding is
{code:java}
class ExecutorResourceRequests() {code}
is a convenience method for the class
{code:java}
class ExecutorResourceRequest(){code}
Class `ExecutorResourceRequests() ` has multiple methods to add different
resources
{code:java}
def offHeapMemory(amount: String): this.type
def memoryOverhead(amount: String): this.type
def memory(amount: String): this.type{code}
and they are stored in a mutable map
{code:java}
new ConcurrentHashMap[String, ExecutorResourceRequest](){code}
The goal of this PR is to offer an API that allows users to build a Request
profile once as shown below
{code:java}
val executorReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder()
.memory("1g") .cores(4) .resource("gpu", 2) .build()
{code}
and then offer ways to add or remove specific resources as shown below
{code:java}
val anotherReqs: ExecutorResourceRequests = ExecutorResourceRequests.builder()
.offHeapMemory("512m") .add(new ExecutorResourceRequest("fpga", 1))
.remove("memory") // This would have no effect as 'memory' wasn't added in this
builder .build()
{code}
Kindly, let me know if my understanding is correct and I would like to work on
this task
> Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have
> functions to remove requests
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-30451
> URL: https://issues.apache.org/jira/browse/SPARK-30451
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Thomas Graves
> Priority: Major
>
> Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have
> functions to remove requests
> Currently in the design ExecutorResourceRequests and TaskREsourceRequests are
> mutable and users can update as they want. It would make sense to add api's
> to remove certain resource requirements from them. This would allow a user to
> create one ExecutorResourceRequests object and then if they want to just
> add/remove something from it they easily could without having to recreate all
> the requests in that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
