ayushi-agarwal commented on PR #11452:
URL:
https://github.com/apache/incubator-gluten/pull/11452#issuecomment-3953962883
@rui-mo I added a log line at the end of destructor also and it also got
printed.
I see the ref count of pool also increases with time for the pool: I have
added these log lines in velox memory manage tryDestructSafeLoop and here in
the end you can see the refcount
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 1048576 allocated
pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0
numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0
reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity
0B freeReservedCapacity 0B]
E20260224 18:18:46.178891 138102 VeloxMemoryManager.cc:345] Attempt to
destruct VeloxMemoryManager failed because there are 4 outstanding memory pools.
I20260224 18:18:46.178897 138102 VeloxMemoryManager.cc:402] There are still
outstanding Velox memory allocations. Waiting for 9700 ms to let possible async
tasks done...
E20260224 18:18:46.183820 138102 VeloxMemoryManager.cc:344] Memory
Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 74.00MB
task.Gluten_Stage_112_TID_24630_VTID_804 usage 0B reserved 0B peak
73.00MB
node.0 usage 0B reserved 0B peak 72.00MB
op.0.0.0.TableScan usage 0B reserved 0B peak 70.72MB
**refcount 2**
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 1048576 allocated
pages 0 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0
numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0
reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity
0B freeReservedCapacity 0B]
E20260224 18:18:46.183887 138102 VeloxMemoryManager.cc:345] Attempt to
destruct VeloxMemoryManager failed because there are 4 outstanding memory pools.
I20260224 18:18:46.183890 138102 VeloxMemoryManager.cc:402] There are still
outstanding Velox memory allocations. Waiting for 14550 ms to let possible
async tasks done...
E20260224 18:18:46.188819 138102 VeloxMemoryManager.cc:408] Failed to
release Velox memory manager after 43350ms as there are still outstanding
memory resources.
E20260224 18:18:46.188876 138102 Exceptions.h:66] Line:
/root/incubator-gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp:153,
Function:~MemoryManager, Expression: pools_.size() != 0 (1 vs 0). There are
unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 0B number of pools
4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
__sys_shared_leaf__0 usage 0B reserved 0B peak 0B
__sys_tracing__ usage 0B reserved 0B peak 0B
__sys_caching__ usage 0B reserved 0B peak 0B
__sys_spilling__ usage 0B reserved 0B peak 0B
root usage 0B reserved 0B peak 74.00MB
task.Gluten_Stage_112_TID_24630_VTID_804 usage 0B reserved 0B peak
73.00MB
node.0 usage 0B reserved 0B peak 72.00MB
op.0.0.0.TableScan usage 0B reserved 0B peak 70.72MB
**refcount 3**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
