Next problem: framework is failing to register. 2014-10-15 22:15:48,024:28619(0x7f852d259700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:2181], sessionId=0x349110433970e3b, negotiated timeout=10000 I1015 22:15:48.024787 28763 group.cpp:313] Group process (group(3)@ 10.88.26.227:34348) connected to ZooKeeper I1015 22:15:48.024837 28763 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I1015 22:15:48.024868 28763 group.cpp:385] Trying to create path '/mesos/master' in ZooKeeper E1015 22:16:48.013 THREAD132 org.apache.aurora.scheduler.SchedulerLifecycle$6$3.run: Framework has not been registered within the tolerated delay. I1015 22:16:48.014 THREAD132 com.twitter.common.util.StateMachine$Builder$1.execute: SchedulerLifecycle state machine transition LEADER_AWAITING_REGISTRATION -> DEAD
My Zookeeper logs have this: 2014-10-15 22:15:47,831 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:41116 2014-10-15 22:15:47,836 - INFO [CommitProcessor:3:ZooKeeperServer@595] - Established session 0x349110433970e3a with negotiated timeout 4000 for client /127.0.0.1:41116 2014-10-15 22:15:48,017 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:41117 2014-10-15 22:15:48,017 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] - Connection request from old client /127.0.0.1:41117; will be dropped if server is in r-o mode 2014-10-15 22:15:48,018 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0.1:41117 2014-10-15 22:15:48,024 - INFO [CommitProcessor:3:ZooKeeperServer@595] - Established session 0x349110433970e3b with negotiated timeout 10000 for client /127.0.0.1:41117 2014-10-15 22:16:48,435 - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x349110433970e39, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2014-10-15 22:16:48,436 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:41114 which had sessionid 0x349110433970e39 I've removed /mesos/master and verified that it gets recreated: [zk: localhost:2181(CONNECTED) 4] rmr /mesos/master [zk: localhost:2181(CONNECTED) 5] ls /mesos/master Node does not exist: /mesos/master [zk: localhost:2181(CONNECTED) 6] get /mesos/master cZxid = 0x10000b717 ctime = Wed Oct 15 23:34:17 UTC 2014 mZxid = 0x10000b717 mtime = Wed Oct 15 23:34:17 UTC 2014 pZxid = 0x10000b717 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 0 numChildren = 0 Anyone know what's going wrong? On Wed, Oct 15, 2014 at 2:47 PM, Dobromir Montauk <dobro...@tellapart.com> wrote: > I found instructions that fixed the current issue here: > > http://wilderness.apache.org/channels/?f=aurora/2014-05-27 > > Would be nice to have a more friendly error message :) > > On Wed, Oct 15, 2014 at 2:22 PM, Dobromir Montauk <dobro...@tellapart.com> > wrote: > >> Hi, >> >> I've brought up Aurora on my Mesos master node with the following command: >> >> ubuntu@ec2-54-82-17-37:~/$ >> >> GLOG_v=2 >> LIBPROCESS_PORT=5050 >> LIBPROCESS_IP=127.0.0.1 >> AURORA_HOME=/usr/local/aurora-scheduler >> DIST_DIR=/home/ubuntu/aurora-scheduler/dist >> AURORA_HOME=/usr/local/aurora-scheduler >> >> sudo /usr/local/aurora-scheduler/bin/aurora-scheduler \ >> -cluster_name=tellapart \ >> -http_port=8081 \ >> -native_log_quorum_size=1 \ >> -zk_endpoints=localhost:2181 \ >> -mesos_master_address=54.166.50.69:5050,54.160.61.169:5050,localhost:5050 >> \ >> -serverset_path=/aurora/scheduler \ >> -native_log_zk_group_path=/aurora/replicated-log \ >> -native_log_file_path=$AURORA_HOME/scheduler/db \ >> -backup_dir=$AURORA_HOME/scheduler/backups \ >> -thermos_executor_path=/dev/null \ >> -gc_executor_path=$DIST_DIR/gc_executor.pex \ >> -enable_beta_updater=true \ >> -vlog=INFO \ >> -logtostderr >> >> Attached is the entire log, but basically I'm seeing this: >> >> I1015 21:18:05.315263 27634 group.cpp:313] Group process (group(1)@ >> 10.88.26.227:40393) connected to ZooKeeper >> I1015 21:18:05.315322 27634 group.cpp:787] Syncing group operations: >> queue size (joins, cancels, datas) = (0, 0, 0) >> I1015 21:18:05.315348 27634 group.cpp:385] Trying to create path >> '/aurora/replicated-log' in ZooKeeper >> I1015 21:18:05.316 THREAD1 >> com.twitter.common.zookeeper.CandidateImpl$4.onGroupChange: Candidate >> /aurora/scheduler/singleton_candidate_0000000008 is now leader of group: >> [singleton_candidate_0000000008] >> I1015 21:18:05.317 THREAD1 >> com.twitter.common.util.StateMachine$Builder$1.execute: SchedulerLifecycle >> state machine transition STORAGE_PREPARED -> LEADER_AWAITING_REGISTRATION >> I1015 21:18:05.317 THREAD1 >> org.apache.aurora.scheduler.SchedulerLifecycle$6.execute: Elected as >> leading scheduler! >> I1015 21:18:05.330394 27634 network.hpp:423] ZooKeeper group memberships >> changed >> I1015 21:18:05.330660 27639 group.cpp:658] Trying to get >> '/aurora/replicated-log/0000000008' in ZooKeeper >> I1015 21:18:05.331550 27635 network.hpp:461] ZooKeeper group PIDs: { >> log-replica(1)@10.88.26.227:40393 } >> I1015 21:18:06.027016 27634 replica.cpp:638] Replica in EMPTY status >> received a broadcasted recover request >> I1015 21:18:06.027216 27634 recover.cpp:188] Received a recover response >> from a replica in EMPTY status >> <repeat last 2 message ad nauseum> >> >> How can I debug what's going on? >> >> Thanks, >> Dobromir >> > >
