Tested on version 2.2.11 (but seems like trunck 3.x is still the same for the
related code path), using nodetool refresh for restoring a snapshot
I guess the Cassandra committers can do something with this 😊.
Calling StorageService.loadNewSSTables function results in deadlock with
compaction background task,
because :
From StorageService class , function public void loadNewSSTables(String ksName,
String cfName)
a call is made to ColumnFamilyStore class , function public static synchronized
void loadNewSSTables(String ksName, String cfName)
and then a call to Keyspace class, function public static Keyspace open(String
keyspaceName)
getting to the function private static Keyspace open(String keyspaceName,
Schema schema, boolean loadSSTables)
finally trying to get a lock by synchronized (Keyspace.class)
So inside the ColumnFamilyStore class lock, there is an attempt to get the lock
on the Keyspace.class
Now at the same time I have the thread OptionalTasks executing the
ColumnFamilyStore.getBackgroundCompactionTaskSubmitter() task.
The thread task is also calling Keyspace.open function, already progressed as
far as getting the lock on Keyspace class.
But then the call also initializes the column families and thus is calling on
class ColumnFamilyStore the public static synchronized ColumnFamilyStore
createColumnFamilyStore ...
So function 1 locks A and then B
And function 2 locks B and then A
leading to deadlock
Regards,
Ignace
Daemon System Thread [RMI TCP Connection(19)-10.x.x.x] (Suspended)
owns: Class<T> (org.apache.cassandra.db.ColumnFamilyStore) (id=2814)
waiting for: Class<T> (org.apache.cassandra.db.Keyspace) (id=3269)
Keyspace.open(String, Schema, boolean) line: 110
Keyspace.open(String) line: 93
ColumnFamilyStore.loadNewSSTables(String, String) line: 736
StorageService.loadNewSSTables(String, String) line: 4378
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not
available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 498
Trampoline.invoke(Method, Object, Object[]) line: 71
GeneratedMethodAccessor38.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 498
MethodUtil.invoke(Method, Object, Object[]) line: 275
StandardMBeanIntrospector.invokeM2(Method, Object, Object[], Object)
line: 112
StandardMBeanIntrospector.invokeM2(Object, Object, Object[], Object)
line: 46
StandardMBeanIntrospector(MBeanIntrospector<M>).invokeM(M, Object,
Object[], Object) line: 237
PerInterface<M>.invoke(Object, String, Object[], String[], Object)
line: 138
StandardMBeanSupport(MBeanSupport<M>).invoke(String, Object[],
String[]) line: 252
DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[],
String[]) line: 819
JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: 801
RMIConnectionImpl.doOperation(int, Object[]) line: 1468
RMIConnectionImpl.access$300(RMIConnectionImpl, int, Object[]) line: 76
RMIConnectionImpl$PrivilegedOperation.run() line: 1309
RMIConnectionImpl.doPrivilegedOperation(int, Object[], Subject) line:
1401
RMIConnectionImpl.invoke(ObjectName, String, MarshalledObject,
String[], Subject) line: 829
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not
available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 498
UnicastServerRef.dispatch(Remote, RemoteCall) line: 357
Transport$1.run() line: 200
Transport$1.run() line: 197
AccessController.doPrivileged(PrivilegedExceptionAction<T>,
AccessControlContext) line: not available [native method]
TCPTransport(Transport).serviceCall(RemoteCall) line: 196
TCPTransport.handleMessages(Connection, boolean) line: 568
TCPTransport$ConnectionHandler.run0() line: 826
TCPTransport$ConnectionHandler.lambda$run$0() line: 683
1690776575.run() line: not available
AccessController.doPrivileged(PrivilegedAction<T>,
AccessControlContext) line: not available [native method]
TCPTransport$ConnectionHandler.run() line: 682
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1149
ThreadPoolExecutor$Worker.run() line: 624
Thread.run() line: 748
Daemon Thread [OptionalTasks:1] (Suspended)
owns: Class<T> (org.apache.cassandra.db.Keyspace) (id=3269)
waiting for: Class<T> (org.apache.cassandra.db.ColumnFamilyStore)
(id=2814)
ColumnFamilyStore.createColumnFamilyStore(Keyspace, String,
IPartitioner, CFMetaData, boolean) line: 532
ColumnFamilyStore.createColumnFamilyStore(Keyspace, CFMetaData,
boolean) line: 522
Keyspace.initCf(CFMetaData, boolean) line: 342
Keyspace.<init>(String, boolean) line: 270
Keyspace.open(String, Schema, boolean) line: 116
Keyspace.open(String) line: 93
Keyspace$1.apply(String) line: 80
Keyspace$1.apply(Object) line: 77
Iterators$7.transform(F) line: 750
Iterators$7(TransformedIterator<F,T>).next() line: 47
ColumnFamilyStore$2.run() line: 262
DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run() line:
118
Executors$RunnableAdapter<T>.call() line: 511
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).runAndReset()
line: 308
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$301(ScheduledThreadPoolExecutor$ScheduledFutureTask)
line: 180
ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: 294
DebuggableScheduledThreadPoolExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker)
line: 1149
ThreadPoolExecutor$Worker.run() line: 624
Thread.run() line: 748
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
