[
https://issues.apache.org/jira/browse/CASSANDRA-14530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ALEXEY KASHINTSEV updated CASSANDRA-14530:
------------------------------------------
Environment:
Description: Debian GNU/Linux 8.5 (jessie)
Codename: jessie
16GB mem. 6 allocated by java only.
Linux XXXX04 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02)
x86_64 GNU/Linux
cassandra 3.11.2
Output of command
XProf file attached below
{code:java}
XXXXXXXXX04:/var/lib/cassandra# nodetool repair archive statuses -full
[2018-06-19 17:17:38,170] Starting repair command #1
(84fd0590-73cb-11e8-9815-6d9cda004dd5), repairing keyspace archive with repair
options (parallelism: parallel, primary range: false, incremental: false, job
threads: 1, ColumnFamilies: [statuses], dataCenters: [], hosts: [], # of
ranges: 967, pull repair: false)
Exception occurred during clean-up.
java.lang.reflect.UndeclaredThrowableException
Cassandra has shutdown.
error: [2018-06-19 17:17:42,233] JMX connection closed. You should check server
log for repair status of keyspace archive(Subsequent keyspaces are not going to
be repaired).
-- StackTrace --
java.io.IOException: [2018-06-19 17:17:42,233] JMX connection closed. You
should check server log for repair status of keyspace archive(Subsequent
keyspaces are not going to be repaired).
at
org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at
javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
at
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at
javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
{code}
was:
Description: Debian GNU/Linux 8.5 (jessie)
Codename: jessie
16GB mem. 6 allocated by java only.
Linux xen-new-cassandra04 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3
(2016-07-02) x86_64 GNU/Linux
cassandra 3.11.2
Output of command
XProf file attached below
{code:java}
XXXXXXXXX04:/var/lib/cassandra# nodetool repair archive statuses -full
[2018-06-19 17:17:38,170] Starting repair command #1
(84fd0590-73cb-11e8-9815-6d9cda004dd5), repairing keyspace archive with repair
options (parallelism: parallel, primary range: false, incremental: false, job
threads: 1, ColumnFamilies: [statuses], dataCenters: [], hosts: [], # of
ranges: 967, pull repair: false)
Exception occurred during clean-up.
java.lang.reflect.UndeclaredThrowableException
Cassandra has shutdown.
error: [2018-06-19 17:17:42,233] JMX connection closed. You should check server
log for repair status of keyspace archive(Subsequent keyspaces are not going to
be repaired).
-- StackTrace --
java.io.IOException: [2018-06-19 17:17:42,233] JMX connection closed. You
should check server log for repair status of keyspace archive(Subsequent
keyspaces are not going to be repaired).
at
org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98)
at
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at
javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
at
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at
javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
{code}
> Unable to start repair process
> ------------------------------
>
> Key: CASSANDRA-14530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14530
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction, Core, Repair
> Environment: Description: Debian GNU/Linux 8.5 (jessie)
> Codename: jessie
> 16GB mem. 6 allocated by java only.
> Linux XXXX04 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02)
> x86_64 GNU/Linux
> cassandra 3.11.2
>
> Output of command
> XProf file attached below
> {code:java}
> XXXXXXXXX04:/var/lib/cassandra# nodetool repair archive statuses -full
> [2018-06-19 17:17:38,170] Starting repair command #1
> (84fd0590-73cb-11e8-9815-6d9cda004dd5), repairing keyspace archive with
> repair options (parallelism: parallel, primary range: false, incremental:
> false, job threads: 1, ColumnFamilies: [statuses], dataCenters: [], hosts:
> [], # of ranges: 967, pull repair: false)
> Exception occurred during clean-up.
> java.lang.reflect.UndeclaredThrowableException
> Cassandra has shutdown.
> error: [2018-06-19 17:17:42,233] JMX connection closed. You should check
> server log for repair status of keyspace archive(Subsequent keyspaces are not
> going to be repaired).
> -- StackTrace --
> java.io.IOException: [2018-06-19 17:17:42,233] JMX connection closed. You
> should check server log for repair status of keyspace archive(Subsequent
> keyspaces are not going to be repaired).
> at
> org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
> at
> javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
> at
> javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
> at
> javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
> at
> javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
> at
> javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
> at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
> at
> javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
> at
> javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> {code}
> Reporter: ALEXEY KASHINTSEV
> Priority: Blocker
> Attachments: hs_err_1529417753.log
>
>
>
> Then repair starts, cassandra goes down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
