Hi, getting weird problem when agent to connect to OpsCenter
OpsCenter installed on VM with DSE and agent.
It's not for production, I have 3 VMs with DSE and OpsCenter for dev/test
purposes.
The stacktrace from agent log is:
vagrant@dsenode03:~$ sudo cat /var/log/datastax-agent/agent.log
Starting DataStax agent monitor datastax_agent_monitor
INFO [main] 2015-04-05 13:32:31,594 Loading conf files:
/var/lib/datastax-agent/conf/address.yaml
INFO [main] 2015-04-05 13:32:31,642 Java vendor/version: Java HotSpot(TM)
64-Bit Server VM/1.7.0_76
INFO [main] 2015-04-05 13:32:31,642 DataStax Agent version: 5.1.1
INFO [main] 2015-04-05 13:32:31,679 Default config values:
{:cassandra_port 9042, :rollups300_ttl 2419200, :settings_cf "settings",
:restore_req_update_period 60, :my_channel_prefix "/agent", :poll_period
60, :thrift_conn_timeout 10000, :rollups60_ttl 604800, :stomp_port 61620,
:shorttime_interval 10, :longtime_interval 300, :max-seconds-to-sleep 25,
:private-conf-props ["initial_token" "listen_address" "broadcast_address"
"rpc_address"], :thrift_port 9160, :async_retry_timeout 5,
:agent-conf-group "global-cluster-agent-group", :jmx_host "127.0.0.1",
:ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1,
:async_queue_size 5000, :backup_staging_dir nil, :read-buffer-size
10000000, :remote_verify_max 300000, :disk_usage_update_period 60,
:throttle-bytes-per-second 500000, :rollups7200_ttl 31536000,
:remote_backup_retries 3, :ssl_keystore nil, :rollup_snapshot_period 300,
:is_package true, :monitor_command
"/usr/share/datastax-agent/bin/datastax_agent_monitor",
:thrift_socket_timeout 5000, :remote_verify_initial_delay 1000,
:cassandra_log_location "/var/log/cassandra/system.log",
:remote_backup_region "us-west-1", :restore_on_transfer_failure false,
:tmp_dir "/var/lib/datastax-agent/tmp/", :config_md5 nil, :jmx_port 7199,
:write-buffer-size 100000, :jmx_metrics_threadpool_size 4, :use_ssl 0,
:rollups86400_ttl 0, :nodedetails_threadpool_size 3, :api_port 61621,
:kerberos_service nil, :backup_file_queue_max 10000, :jmx_thread_pool_size
5, :production 1, :runs_sudo 1, :max_file_transfer_attempts 30,
:stomp_interface nil, :storage_keyspace "OpsCenter", :hosts ["127.0.0.1"],
:rollup_snapshot_threshold 300, :jmx_retry_timeout 30, :unthrottled-default
10000000000, :remote_backup_retry_delay 5000, :remote_backup_timeout 1000,
:seconds-to-read-kill-channel 0.005, :realtime_interval 5, :pdps_ttl 259200}
INFO [main] 2015-04-05 13:32:31,924 Waiting for the config from OpsCenter
INFO [main] 2015-04-05 13:32:31,925 Attempting to determine Cassandra's
broadcast address through JMX
INFO [Initialization] 2015-04-05 13:32:31,926 New JMX connection (
127.0.0.1:7199)
INFO [main] 2015-04-05 13:32:31,947 Starting Jetty server: {:join? false,
:ssl? false, :host nil, :port 61621}
INFO [Jetty] 2015-04-05 13:32:32,026 Jetty server started
INFO [Initialization] 2015-04-05 13:32:32,054 Using 192.168.56.30 as the
cassandra broadcast address
INFO [Initialization] 2015-04-05 13:32:32,135 cassandra RPC address is nil
INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC address is
192.168.56.30
INFO [Initialization] 2015-04-05 13:32:32,135 agent RPC broadcast address
is 192.168.56.30
ERROR [Initialization] 2015-04-05 13:32:32,342 Can't connect to Cassandra,
retrying
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
tried for query failed (tried: /127.0.0.1:9042
(com.datastax.driver.core.TransportException: [/127.0.0.1:9042] Cannot
connect))
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:220)
at
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1231)
at com.datastax.driver.core.Cluster.init(Cluster.java:158)
at com.datastax.driver.core.Cluster.connect(Cluster.java:246)
at clojurewerkz.cassaforte.client$connect_or_close.doInvoke(client.clj:149)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojurewerkz.cassaforte.client$connect.invoke(client.clj:165)
at opsagent.cassandra$setup_cassandra$fn__2491.invoke(cassandra.clj:269)
at again.core$with_retries_STAR_$fn__2363.invoke(core.clj:98)
at again.core$with_retries_STAR_.invoke(core.clj:97)
at opsagent.cassandra$setup_cassandra.invoke(cassandra.clj:267)
at opsagent.opsagent$setup_cassandra.invoke(opsagent.clj:152)
at opsagent.jmx$determine_ip.invoke(jmx.clj:276)
at opsagent.jmx$setup_jmx$fn__2867.invoke(jmx.clj:293)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
And it tries to repeat in several times.
I did:
vagrant@dsenode03:~$ sudo netstat -alnpt | grep 9042
tcp 0 0 192.168.56.30:*9042* 0.0.0.0:*
LISTEN 5490/java
and see that some stuff already listening to this port
# cut some output
vagrant@dsenode03:~$ sudo lsof -p 5490
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
NAME
java 5490 cassandra cwd DIR 252,0 4096 3670233
/usr/share/dse
java 5490 cassandra rtd DIR 252,0 4096 2 /
# cut some output
vagrant@dsenode03:~$ sudo lsof -p 5490 | grep .log
java 5490 cassandra mem REG 252,0 33554432 3803832
/var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370080*.log*
java 5490 cassandra mem REG 252,0 33554432 3803355
/var/lib/cassandra/commi*tlog*/CommitLog-3-1428238370079*.log*
java 5490 cassandra DEL REG 252,0 3670465
/usr/share/dse/common*/log*4j-1.2.17.jar
java 5490 cassandra DEL REG 252,0 3670467
/usr/share/dse/common/slf4j*-log*4j12-1.7.2.jar
java 5490 cassandra 439w REG 252,0 120225 3802518
/var*/log*/cassandra/system*.log*
Cassandra DSE did occupy this port. What do I do wrong???
I have three nodes and agents work there witout any problems
