ok I copied the cassandra.env from the host that had cassandra listening on
port 7199 to the node that wasn't.
That got it listening on the JMX port:
[root@beta:~] #lsof -i :7199
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 9197 root 45u IPv4 6411278 0t0 TCP *:7199 (LISTEN)
But even tho I can telnet to that port from the seed node:
[root@beta-new:~] #telnet 10.10.1.18 7199
Trying 166.78.27.18...
Connected to 166.78.27.18.
Escape character is '^]'.
I still get connection refused when trying to decommission the node:
[root@beta-new:~] #nodetool -host 10.10.1.18 decommission
Failed to connect to '166.78.27.18:7199': Connection refused
To Colin, thanks for the information!
nodetool decomission <host-id-of-node-to-decomission>
UN means UP, NORMAL
Oh and thanks I tried that and it seems to be working!
[root@beta-new:~] #nodetool decommission
f2a48fc7-a362-43f5-9061-4bb3739fdeaf
Decommission will decommission the node you are connected to and does not
take arguments!
Sorry guys, wrote this email in a hurry as we're checking out of a hotel
room currently. :) I'll let you know if this does work.
On Sun, May 25, 2014 at 10:19 AM, Colin Clark <co...@clark.ws> wrote:
> Try this:
>
> nodetool decomission <host-id-of-node-to-decomission>
>
> UN means UP, NORMAL
>
> --
> Colin
> +1 320 221 9531
>
>
>
> On Sun, May 25, 2014 at 9:09 AM, Tim Dunphy <bluethu...@gmail.com> wrote:
>
>> Also for information that may help diagnose this issue I am running
>> cassandra 2.0.7
>>
>> I am also using these java options:
>>
>> [root@beta:/etc/alternatives/cassandrahome] #grep -i jvm_opts
>> conf/cassandra-env.sh | grep -v '#'
>> JVM_OPTS="$JVM_OPTS -ea"
>> JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar"
>> JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
>> JVM_OPTS="$JVM_OPTS -XX:+UseThreadPriorities"
>> JVM_OPTS="$JVM_OPTS -XX:ThreadPriorityPolicy=42"
>> JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
>> JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
>> JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
>> JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
>> JVM_OPTS="$JVM_OPTS
>> -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
>> JVM_OPTS="$JVM_OPTS -Xss256k"
>> JVM_OPTS="$JVM_OPTS -XX:StringTableSize=1000003"
>> JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
>> JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
>> JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
>> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
>> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
>> JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75"
>> JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
>> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
>> JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
>> JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>> JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
>>
>>
>> Still need to figure out why the node I want to decommission isn't
>> listening on port 7199 and how I can actually decommission it.
>>
>> Thanks
>> Tim
>>
>>
>> On Sun, May 25, 2014 at 9:20 AM, Tim Dunphy <bluethu...@gmail.com> wrote:
>>
>>>
>>> Hey all,
>>>
>>> I'm attempting to decommission a node I want to remove.
>>>
>>> First I get a status of the ring
>>>
>>> [root@beta-new:~] #nodetool status
>>>
>>> Datacenter: datacenter1
>>>
>>> =======================
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> -- Address Load Tokens Owns Host ID
>>> Rack
>>>
>>> UN 10.10.1.94 197.37 KB 256 49.4%
>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e rack1
>>>
>>> UN 10.10.1.18 216.95 KB 256 50.6%
>>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf rack
>>>
>>>
>>> I see that the node I want to remove is UP. Tho I believe UN means up I
>>> don't know what it stands for.
>>>
>>>
>>> [root@beta-new:~] #nodetool -host 10.10.1.18 decommission
>>>
>>> Failed to connect to ' 10.10.1.18 : Connection timed out
>>>
>>> The connection to the node I want to decommission times out. :(
>>>
>>> I’m running this node from the seed node, and while I do see port 7199
>>> active and listening there, I do NOT see this port active and listening on
>>> the node that I want to decommission.
>>>
>>>
>>> Seed node:
>>>
>>> [root@beta-new:~] #lsof -i :7199
>>>
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>
>>> java 15331 root 51u IPv4 566368606 0t0 TCP *:7199 (LISTEN)
>>>
>>>
>>> [root@beta:/etc/alternatives/cassandrahome] #lsof -i :7199
>>>
>>> [root@beta:/etc/alternatives/cassandrahome] #
>>>
>>>
>>> However cassandra does seem to be running on the node I want to
>>> decommission in addition to it being shown as UN by nodetool status:
>>>
>>>
>>> [root@beta:/etc/alternatives/cassandrahome] #netstat -tulpn | grep -i
>>> listen | grep java
>>>
>>> tcp 0 0 0.0.0.0:46755 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 10.10.1.18:9160 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 0.0.0.0:42990 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 10.10.1.18:8081 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 10.10.1.18:9042 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 10.10.1.18:7000 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>> tcp 0 0 0.0.0.0:7198 0.0.0.0:*
>>> LISTEN 23039/java
>>>
>>>
>>> So why do you think my seed is listening on port 7199 but the node I
>>> want to get rid of is not? And how can I accomplish my goal of deleting the
>>> unwanted node?
>>>
>>>
>>> Thanks
>>>
>>> Tim
>>>
>>>
>>>
>>> --
>>> GPG me!!
>>>
>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>
>>>
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
