>
> In case the above doesn't work, another thing to be aware of is that JMX
> uses 2 different ports. The initial connection to 7199 causes a second port
> to be opened, which is normally assigned randomly to an available and
> otherwise unused port above 1024. If your server has a
> firewall/ACL/Security Group configured, it probably doesn't expose the port.
>
> Assuming you are on Java 7 (u4 or later), you can add the following to
> your environment:
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7200"
>
> to force a fixed port (7200 in this case) for the second one, which is
> more firewall friendly. You'll still need to be sure access to port 7200 is
> allowed, but at least it's only a single port.
>
Hope that helps,
>
Hi Steven,
Thank you for that. I'll give this a try. Right now I am down to one
cassandra node with one node reported down in nodetool stats. I'll probably
make another attempt to revive another cassandra node in order to get rid
of the lingering dead one later on this week.
But in the meantime, as an FYI I am on java 8 currently. So this should
hopefully work.
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
> Thanks
>
Tim
On Tue, May 27, 2014 at 12:04 PM, Steven A Robenalt
<srobe...@stanford.edu>wrote:
> Hi Tim,
>
> In case the above doesn't work, another thing to be aware of is that JMX
> uses 2 different ports. The initial connection to 7199 causes a second port
> to be opened, which is normally assigned randomly to an available and
> otherwise unused port above 1024. If your server has a
> firewall/ACL/Security Group configured, it probably doesn't expose the port.
>
> Assuming you are on Java 7 (u4 or later), you can add the following to
> your environment:
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=7200"
>
> to force a fixed port (7200 in this case) for the second one, which is
> more firewall friendly. You'll still need to be sure access to port 7200 is
> allowed, but at least it's only a single port.
>
> Hope that helps,
>
> Steve
>
>
>
> On Sun, May 25, 2014 at 7:48 AM, Tim Dunphy <bluethu...@gmail.com> wrote:
>
>> ok I copied the cassandra.env from the host that had cassandra listening
>> on port 7199 to the node that wasn't.
>>
>> That got it listening on the JMX port:
>>
>> [root@beta:~] #lsof -i :7199
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 9197 root 45u IPv4 6411278 0t0 TCP *:7199 (LISTEN)
>>
>> But even tho I can telnet to that port from the seed node:
>>
>> [root@beta-new:~] #telnet 10.10.1.18 7199
>> Trying 166.78.27.18...
>> Connected to 166.78.27.18.
>> Escape character is '^]'.
>>
>>
>> I still get connection refused when trying to decommission the node:
>>
>> [root@beta-new:~] #nodetool -host 10.10.1.18 decommission
>> Failed to connect to '166.78.27.18:7199': Connection refused
>>
>> To Colin, thanks for the information!
>>
>> nodetool decomission <host-id-of-node-to-decomission>
>>
>> UN means UP, NORMAL
>>
>> Oh and thanks I tried that and it seems to be working!
>>
>> [root@beta-new:~] #nodetool decommission
>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf
>> Decommission will decommission the node you are connected to and does not
>> take arguments!
>>
>> Sorry guys, wrote this email in a hurry as we're checking out of a hotel
>> room currently. :) I'll let you know if this does work.
>>
>>
>>
>> On Sun, May 25, 2014 at 10:19 AM, Colin Clark <co...@clark.ws> wrote:
>>
>>> Try this:
>>>
>>> nodetool decomission <host-id-of-node-to-decomission>
>>>
>>> UN means UP, NORMAL
>>>
>>> --
>>> Colin
>>> +1 320 221 9531
>>>
>>>
>>>
>>> On Sun, May 25, 2014 at 9:09 AM, Tim Dunphy <bluethu...@gmail.com>wrote:
>>>
>>>> Also for information that may help diagnose this issue I am running
>>>> cassandra 2.0.7
>>>>
>>>> I am also using these java options:
>>>>
>>>> [root@beta:/etc/alternatives/cassandrahome] #grep -i jvm_opts
>>>> conf/cassandra-env.sh | grep -v '#'
>>>> JVM_OPTS="$JVM_OPTS -ea"
>>>> JVM_OPTS="$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar"
>>>> JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseThreadPriorities"
>>>> JVM_OPTS="$JVM_OPTS -XX:ThreadPriorityPolicy=42"
>>>> JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
>>>> JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
>>>> JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
>>>> JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
>>>> JVM_OPTS="$JVM_OPTS
>>>> -XX:HeapDumpPath=$CASSANDRA_HEAPDUMP_DIR/cassandra-`date +%s`-pid$$.hprof"
>>>> JVM_OPTS="$JVM_OPTS -Xss256k"
>>>> JVM_OPTS="$JVM_OPTS -XX:StringTableSize=1000003"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
>>>> JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
>>>> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
>>>> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
>>>> JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
>>>> JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
>>>> JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>>> JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
>>>>
>>>>
>>>> Still need to figure out why the node I want to decommission isn't
>>>> listening on port 7199 and how I can actually decommission it.
>>>>
>>>> Thanks
>>>> Tim
>>>>
>>>>
>>>> On Sun, May 25, 2014 at 9:20 AM, Tim Dunphy <bluethu...@gmail.com>wrote:
>>>>
>>>>>
>>>>> Hey all,
>>>>>
>>>>> I'm attempting to decommission a node I want to remove.
>>>>>
>>>>> First I get a status of the ring
>>>>>
>>>>> [root@beta-new:~] #nodetool status
>>>>>
>>>>> Datacenter: datacenter1
>>>>>
>>>>> =======================
>>>>>
>>>>> Status=Up/Down
>>>>>
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>
>>>>> -- Address Load Tokens Owns Host ID
>>>>> Rack
>>>>>
>>>>> UN 10.10.1.94 197.37 KB 256 49.4%
>>>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e rack1
>>>>>
>>>>> UN 10.10.1.18 216.95 KB 256 50.6%
>>>>> f2a48fc7-a362-43f5-9061-4bb3739fdeaf rack
>>>>>
>>>>>
>>>>> I see that the node I want to remove is UP. Tho I believe UN means up
>>>>> I don't know what it stands for.
>>>>>
>>>>>
>>>>> [root@beta-new:~] #nodetool -host 10.10.1.18 decommission
>>>>>
>>>>> Failed to connect to ' 10.10.1.18 : Connection timed out
>>>>>
>>>>> The connection to the node I want to decommission times out. :(
>>>>>
>>>>> I’m running this node from the seed node, and while I do see port 7199
>>>>> active and listening there, I do NOT see this port active and listening on
>>>>> the node that I want to decommission.
>>>>>
>>>>>
>>>>> Seed node:
>>>>>
>>>>> [root@beta-new:~] #lsof -i :7199
>>>>>
>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>>>
>>>>> java 15331 root 51u IPv4 566368606 0t0 TCP *:7199 (LISTEN)
>>>>>
>>>>>
>>>>> [root@beta:/etc/alternatives/cassandrahome] #lsof -i :7199
>>>>>
>>>>> [root@beta:/etc/alternatives/cassandrahome] #
>>>>>
>>>>>
>>>>> However cassandra does seem to be running on the node I want to
>>>>> decommission in addition to it being shown as UN by nodetool status:
>>>>>
>>>>>
>>>>> [root@beta:/etc/alternatives/cassandrahome] #netstat -tulpn | grep -i
>>>>> listen | grep java
>>>>>
>>>>> tcp 0 0 0.0.0.0:46755 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 10.10.1.18:9160 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 0.0.0.0:42990 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 10.10.1.18:8081 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 10.10.1.18:9042 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 10.10.1.18:7000 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>> tcp 0 0 0.0.0.0:7198 0.0.0.0:*
>>>>> LISTEN 23039/java
>>>>>
>>>>>
>>>>> So why do you think my seed is listening on port 7199 but the node I
>>>>> want to get rid of is not? And how can I accomplish my goal of deleting
>>>>> the
>>>>> unwanted node?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Tim
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> GPG me!!
>>>>>
>>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> GPG me!!
>>>>
>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>>
>>>>
>>>
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>
>
> --
> Steve Robenalt
> Software Architect
> HighWire | Stanford University
> 425 Broadway St, Redwood City, CA 94063
>
> srobe...@stanford.edu
> http://highwire.stanford.edu
>
>
>
>
>
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
