Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49492: URL: https://github.com/apache/spark/pull/49492#issuecomment-2594089464 Merged to master/4.0 for Apache Spark 4.0.0. Thank you again, @pan3793 and @parthchandra . -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun closed pull request #49492: [SPARK-50819] Refactor Spark profiler module URL: https://github.com/apache/spark/pull/49492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on code in PR #49492: URL: https://github.com/apache/spark/pull/49492#discussion_r1917416229 ## connector/profiler/README.md: ## @@ -39,7 +39,7 @@ For more information on async_profiler see the [Async Profiler Manual](https://k To enable code profiling,

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
parthchandra commented on code in PR #49492: URL: https://github.com/apache/spark/pull/49492#discussion_r1917373680 ## connector/profiler/README.md: ## @@ -39,7 +39,7 @@ For more information on async_profiler see the [Async Profiler Manual](https://k To enable code profiling,

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
pan3793 commented on PR #49492: URL: https://github.com/apache/spark/pull/49492#issuecomment-2593031373 With SPARK-50810, the profiler module compilation is covered by regular [PR's CI](https://github.com/pan3793/spark/actions/runs/12788664164/job/35650454602), I also verified it in an int

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-14 Thread via GitHub
dongjoon-hyun commented on PR #49492: URL: https://github.com/apache/spark/pull/49492#issuecomment-2591609919 Sure, I'd love to help. We need a real test in production environment for this patch, doesn't it, @pan3793 ? For today, the community is a little busy for creating `branch-4.0

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-14 Thread via GitHub
pan3793 commented on PR #49492: URL: https://github.com/apache/spark/pull/49492#issuecomment-2591582381 @dongjoon-hyun @parthchandra @LuciferYang would you please take a look at this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-14 Thread via GitHub
pan3793 commented on code in PR #49492: URL: https://github.com/apache/spark/pull/49492#discussion_r1915902919 ## connector/profiler/README.md: ## @@ -60,36 +68,28 @@ Then enable the profiling in the configuration. 4.0.0 - spark.executor.profiling.localDir + spark.profi

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-14 Thread via GitHub
pan3793 commented on code in PR #49492: URL: https://github.com/apache/spark/pull/49492#discussion_r1915902396 ## connector/profiler/README.md: ## @@ -60,36 +68,28 @@ Then enable the profiling in the configuration. 4.0.0 - spark.executor.profiling.localDir + spark.profi

[PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-14 Thread via GitHub
pan3793 opened a new pull request, #49492: URL: https://github.com/apache/spark/pull/49492 ### What changes were proposed in this pull request? This PR refactors the `ExecutorProfilerPlugin` to prepare to support driver profiling, it - re-organize the classes from `o.a