[
https://issues.apache.org/jira/browse/CASSANDRA-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18045157#comment-18045157
]
Stefan Miklosovic commented on CASSANDRA-21074:
-----------------------------------------------
Yeah, for some reason, this is happening:
{code}
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:250 -
LEAK DETECTED: a reference (class
org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary)
to class
org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary
was not released before the reference was garbage collected
[junit-timeout] ERROR [Reference-Reaper] 2025-12-15T12:24:48,516 Ref.java:290 -
Allocate trace class
org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1271562936:ZstdCompressionDictionary:
[junit-timeout] Thread[ScheduledTasks:1,5,ScheduledTasks]
[junit-timeout] at
java.base/java.lang.Thread.getStackTrace(Thread.java:1619)
[junit-timeout] at
org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:280)
[junit-timeout] at
org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:201)
[junit-timeout] at
org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:118)
[junit-timeout] at
org.apache.cassandra.db.compression.ZstdCompressionDictionary.<init>(ZstdCompressionDictionary.java:61)
[junit-timeout] at
org.apache.cassandra.db.compression.CompressionDictionary$Kind$1.createDictionary(CompressionDictionary.java:309)
[junit-timeout] at
org.apache.cassandra.db.compression.CompressionDictionary.createFromRow(CompressionDictionary.java:282)
[junit-timeout] at
org.apache.cassandra.schema.SystemDistributedKeyspace.retrieveLatestCompressionDictionary(SystemDistributedKeyspace.java:452)
[junit-timeout] at
org.apache.cassandra.db.compression.CompressionDictionaryScheduler.refreshDictionaryFromSystemTable(CompressionDictionaryScheduler.java:137)
[junit-timeout] at
org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:138)
[junit-timeout] at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[junit-timeout] at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[junit-timeout] at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[junit-timeout] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at java.base/java.lang.Thread.run(Thread.java:840)
{code}
I have added a patch here (1) which closes dictionary before we return. That
will address the issue Dmitry has filed. I think what happens is that we create
"new Ref" in {{ZstdCompressionDictionary}}'s constructor then we populate
{{CompositeData}} in
{{CompressionDictionaryManager.getCompressionDictionary()}}, we return it, but
we just kept that {{ZstdCompressionDictionary}} in a limbo, we never released
it anywhere. Then it will be GC'ed but nobody released it. That is what that
error message is about, I believe.
However, same is kind of happening when we go to do
"refreshDictionaryFromSystemTable", we are not releasing anywhere when that
dictionary instance is already in that cache, no? If it is in cache, we just
silently throw that instance away but we are not releasing anywhere. If that
instance was indeed added to the cache then it would be released when cache is
invalidated or similar.
I think that helped to not fail it as this ticket did, (2), there is some other
flakiness though I am investigating.
I still see above stacktrace I posted in the logs every now and then though.
That is still not resolved.
(1) https://github.com/apache/cassandra/pull/4521
(2)
https://app.circleci.com/pipelines/github/instaclustr/cassandra/6167/workflows/d2f6d2e0-db67-4c9f-a1d5-514cbea5e2cb/jobs/465458/tests
> Flaky test -
> ExportImportListCompressionDictionaryTest.testExportImportListCompressionDictionary
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21074
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21074
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Dmitry Konstantinov
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/2353/testReport/junit/org.apache.cas[…]stExportImportListCompressionDictionary_cdc_jdk11_x86_64/|https://ci-cassandra.apache.org/job/Cassandra-trunk/2353/testReport/junit/org.apache.cassandra.tools.nodetool/ExportImportListCompressionDictionaryTest/Tests___test_cdc_jdk11_11_20___testExportImportListCompressionDictionary_cdc_jdk11_x86_64/]
> There is also a suspicious leak report in the test logs:
> {code:java}
> ERROR [Reference-Reaper] 2025-12-12T20:31:35,396 Ref.java:250 - LEAK
> DETECTED: a reference (class
> org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary)
> to class
> org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary
> was not released before the reference was garbage collected ERROR
> [Reference-Reaper] 2025-12-12T20:31:35,397 Ref.java:290 - Allocate trace
> class
> org.apache.cassandra.db.compression.ZstdCompressionDictionary$Tidy@1218524978:ZstdCompressionDictionary:
> Thread[RMI TCP Connection(52)-127.0.0.1,5,RMI Runtime] at
> java.base/java.lang.Thread.getStackTrace(Thread.java:1602) at
> org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:280) at
> org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:201) at
> org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:118) at
> org.apache.cassandra.db.compression.ZstdCompressionDictionary.<init>(ZstdCompressionDictionary.java:61)
> at
> org.apache.cassandra.db.compression.CompressionDictionary$Kind$1.createDictionary(CompressionDictionary.java:309)
> at
> org.apache.cassandra.db.compression.CompressionDictionary.createFromRow(CompressionDictionary.java:282)
> at
> org.apache.cassandra.schema.SystemDistributedKeyspace.retrieveLatestCompressionDictionary(SystemDistributedKeyspace.java:452)
> at
> org.apache.cassandra.db.compression.CompressionDictionaryManager.getCompressionDictionary(CompressionDictionaryManager.java:282)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at
> jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260) at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at
> java.management/com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
> at
> java.management/com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
> at
> java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:641)
> at
> java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
> at java.base/jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown
> Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> org.apache.cassandra.audit.AuditLogManager$JmxHandler.invoke(AuditLogManager.java:502)
> at com.sun.proxy.$Proxy28.getAttribute(Unknown Source) at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1443)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
> at
> java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:637)
> at java.base/jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown
> Source) at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
> at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) at
> java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) at
> java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
> at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
