Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-07 Thread via GitHub
mbutrovich commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2858775592 Not the end of the world, but this introduces warnings on macOS when using jemalloc: ``` | use tikv_jemalloc_ctl::{epoch, stats}; | ^ ^^^

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-05 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2851473789 Thanks for the reviews @comphead @kazuyukitanimura and @mbutrovich (offline review). I'm going to go ahead and merge this and then iterate on improving it once I have used it

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-05 Thread via GitHub
andygrove merged PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-03 Thread via GitHub
andygrove commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2072417946 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retriev

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-02 Thread via GitHub
comphead commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2072183808 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retrieve

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-02 Thread via GitHub
comphead commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2072182045 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retrieve

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-02 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2847894878 I'm not sure how to reconcile the different resident numbers: ``` JVM_MEMORY: { heapUsed: 1290, heapCommitted: 3184, nonHeapUsed: 111, nonHeapCommitted: 120 } NATI

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-02 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2847888664 I added some `jemalloc` specific logging: ``` NATIVE_MEMORY_JEMALLOC: { allocated=561605760, resident=3749654528 } NATIVE_MEMORY_JEMALLOC: { allocated=360603312, re

Re: [PR] perf: Add memory profiling [datafusion-comet]

2025-05-02 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2847882035 Moving to draft while I experiment with jemalloc-specific logging -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git