Please refer replies embedded below. Thanks in anticipation,
Joy On 18 April 2014 18:18, Michael Shuler <mich...@pbandjelly.org wrote: On 04/18/2014 02:33 AM, Joyabrata Das wrote: Let me explain sequentially. I’ve four nodes in cluster; node1, node2 (seed), node3, node4. 1. Reinstalling 32-Bit Ubuntu problem didn’t solve the problem. *The suggestion here was not to reinstall 32-bit, but to install 64-bit so that your servers were all the same OS, taking that variable out of the equation.* *>** Other three servers were 32-Bit only, so reinstalled the same in Node4* After explicitly enabling port 9160 by ufw command (Uncomplicated FireWall in Ubuntu), node3 joined the ring though no firewall was set among them. Why only node3 required the modification while other three three nodes were already joined the ring without that? *Because your servers have different software installed and enabled by default, like ufw? Dropping *all* firewall rules on *all* servers would be a good starting point to taking this firewall variable out of the equation. **>* * Same is done via command $ ufw default allow Also I’m not sure about other software except Java bit version was different which is now similar* 2. Now opscenter (http://node2:8888/opscenter/index.html) is detecting all nodes in cluster though agent wasn’t getting installed with error "Unable to find a java executable! Please install java or set JAVA_HOME. Exit code: 4", I’m sudoing to install agents. Then I enabled port 61620, 61621 & agent got installed in node3 but not in other nodes, please refer opscenter list view below. *DATACENTER HOSTNAME TOKEN STATUS LOAD (CPU) DATA SIZE* datacenter1 node1 -7564491331177403445 Active - datacenter1 node2 0 Active - datacenter1 node4 -5882736283116946676 Active - datacenter1 node4 5596871268197106065 Active 0 7.1 MB *Oops. You have 2 node4 servers. **>**That was a typo* Why opscenter agent installed only in node3 & not in other nodes from opscenter GUI? *DETAILS REQUESTED* *Node1* *Node3* *echo $JAVA_HOME* /app/jdk1.6.0_16 /app/jdk1.6.0_16 *echo $PATH* /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin *which java* /app/jdk1.6.0_16/bin/java /app/jdk1.6.0_16/bin/java *java -version* java version "1.6.0_16" Java(TM) SE Runtime Environment (build 1.6.0_16-b01) Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode) java version "1.6.0_16" Java(TM) SE Runtime Environment (build 1.6.0_16-b01) Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode) *whoami (as there are diff sudo users from different nodes, I'm trying to install agents one by one in each nodes)* node1 node3 *ps aux | grep cassandra* apacas 17440 4.3 30.1 1377336 1196792 ? Sl Mar19 1868:18 /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar org.apache.cassandra.service.CassandraDaemon apacas 22642 1.5 25.4 1390740 1008932 ? Sl Apr15 66:28 /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache-cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar -Dcassandra.join_ring=true org.apache.cassandra.service.CassandraDaemon 1) how do you start Cassandra on the first 3 nodes? – using /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script 2) how do you start it on the 4th node? – using /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script 3) how did you start opscenter? – using /app/apache/cassandra/opscenter-4.1.1/bin/opscenter script. On 18 April 2014 18:18, Michael Shuler <mich...@pbandjelly.org> wrote: > On 04/18/2014 02:33 AM, Joyabrata Das wrote: > >> Let me explain sequentially. >> > >> > I’ve four nodes in cluster; node1, node2 (seed), node3, node4. >> > >> > 1. Reinstalling 32-Bit Ubuntu problem didn’t solve the problem. >> > > The suggestion here was not to reinstall 32-bit, but to install 64-bit so > that your servers were all the same OS, taking that variable out of the > equation. > > > > After explicitly enabling port 9160 by ufw command (Uncomplicated >> FireWall in Ubuntu), node3 joined the ring though no firewall was set >> among them. >> > >> > Why only node3 required the modification while other three three >> nodes were already joined the ring without that? >> > > Because your servers have different software installed and enabled by > default, like ufw? Dropping *all* firewall rules on *all* servers would be > a good starting point to taking this firewall variable out of the equation. > > > > >> > 2. Now opscenter (http://node2:8888/opscenter/index.html) is >> detecting all nodes in cluster though agent wasn’t getting installed >> with error "Unable to find a java executable! Please install java or set >> JAVA_HOME. Exit code: 4", I’m sudoing to install agents. >> > >> > Then I enabled port 61620, 61621 & agent got installed in node3 but >> not in other nodes, please refer opscenter list view below. >> > >> > DATACENTER HOSTNAME TOKEN >> STATUS LOAD (CPU) DATA >> SIZE >> > >> > datacenter1 node1 >> -7564491331177403445 Active - >> > >> > datacenter1 node2 >> 0 Active - >> > >> > datacenter1 node4 >> -5882736283116946676 Active - >> > >> > datacenter1 node4 >> 5596871268197106065 Active 0 >> 7.1 MB >> > > Oops. You have 2 node4 servers. > > > > >> > Why opscenter agent installed only in node3 & not in other nodes from >> opscenter GUI? >> > >> > DETAILS REQUESTED >> > >> > >> > >> > Node1 >> > >> > Node3 >> > >> > echo $JAVA_HOME >> > >> > /app/jdk1.6.0_16 >> > >> > /app/jdk1.6.0_16 >> > >> > echo $PATH >> > >> > >> >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin >> > >> > >> >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/jdk1.6.0_16/bin >> > >> > which java >> > >> > /app/jdk1.6.0_16/bin/java >> > >> > /app/jdk1.6.0_16/bin/java >> > >> > java -version >> > >> > java version "1.6.0_16" >> > Java(TM) SE Runtime Environment (build 1.6.0_16-b01) >> > Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode) >> > >> > java version "1.6.0_16" >> > Java(TM) SE Runtime Environment (build 1.6.0_16-b01) >> > Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode) >> > >> > whoami (as there are diff sudo users from different nodes, I'm trying >> to install agents one by one in each nodes) >> > > So why are you running cassandra and opscenter as various users, instead > of following the documentation and using the user "cassandra" on all > servers? > > > > >> > node1 >> > >> > node3 >> > >> > ps aux | grep cassandra >> > >> > apacas 17440 4.3 30.1 1377336 1196792 ? Sl Mar19 1868:18 >> /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar >> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M >> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled >> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 >> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly >> -XX:+UseTLAB -Djava.net.preferIPv4Stack=true >> -Dcom.sun.management.jmxremote.port=7199 >> -Dcom.sun.management.jmxremote.ssl=false >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dlog4j.configuration=log4j-server.properties >> -Dlog4j.defaultInitOverride=true -cp >> >> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache >> >> cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar >> org.apache.cassandra.service.CassandraDaemon >> > >> > apacas 22642 1.5 25.4 1390740 1008932 ? Sl Apr15 66:28 >> /app/jdk1.6.0_16/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar >> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M >> -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled >> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 >> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly >> -XX:+UseTLAB -Djava.net.preferIPv4Stack=true >> -Dcom.sun.management.jmxremote.port=7199 >> -Dcom.sun.management.jmxremote.ssl=false >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dlog4j.configuration=log4j-server.properties >> -Dlog4j.defaultInitOverride=true -cp >> >> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-1.2.8.jar:./../lib/apache-cassandra-clientutil-1.2.8.jar:./../lib/apache-cassandra-thrift-1.2.8.jar:./../lib/avro-1.4.0-fixes.jar:./../lib/avro-1.4.0-sources-fixes.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.7.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.6.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree >> > - > 0.1.jar > >> -Dcassandra.join_ring=true org.apache.cassandra.service.CassandraDaemon >> > >> > >> > >> > 1) how do you start Cassandra on the first 3 nodes? – using >> /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script >> > >> > >> > >> > 2) how do you start it on the 4th node? - using >> /app/apache/cassandra/apache-cassandra-1.2.8/bin/cassandra script >> > >> > >> > >> > 3) how did you start opscenter? – using >> /app/apache/cassandra/opscenter-4.1.1/bin/opscenter script. >> > >> > ---------- Forwarded message ---------- >> > From: "Adrian Rodriguez" <ogl...@gmail.com <mailto:ogl...@gmail.com>> >> > Date: Apr 18, 2014 11:50 AM >> > Subject: Re: Apache cassandra not joining cluster ring >> > To: <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> >> > Cc: <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> >> > >> > If I were you, I would make sure all nodes were the same. The 32 / 64 >> thing might not be an issue, but you probably don't want that on a >> production cluster. What would help is if your pasted the output of all >> nodes for these commands: >> > >> > >> > >> > 1) echo $JAVA_HOME >> > >> > 2) echo $PATH >> > >> > 3) which java >> > >> > 4) java -version >> > >> > 5) whoami >> > >> > 6) ps aux | grep cassandra >> > >> > >> > >> > And if you could answer these questions: >> > >> > >> > >> > 1) how do you start Cassandra on the first 3 nodes? >> > >> > >> > >> > 2) how do you start it on the 4th node? >> > >> > >> > >> > 3) how did you start opscenter? >> > >> > >> > >> > I'm also a bit confused as to what your current problem is. Are you >> having trouble installing the opscener agents from the opscener ui? >> > > -- > Kind regards, > Michael > >
