[
https://issues.apache.org/jira/browse/FLINK-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067189#comment-16067189
]
Scott Kidder commented on FLINK-7021:
-------------------------------------
Logs from Flink Task Manager during startup when an unresolvable Zookeeper
hostname is given:
{noformat}
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: recovery.zookeeper.quorum, foo.bar:2181
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: recovery.zookeeper.storageDir,
hdfs://hdfs:8020/flink/recovery
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: recovery.jobmanager.port, 6123
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.rpc.port, 6122
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.data.port, 6121
2017-06-28 20:28:22,674 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.hostname, 10.2.45.10
2017-06-28 20:28:24,249 INFO org.apache.flink.runtime.blob.FileSystemBlobStore
- Creating highly available BLOB storage directory at
hdfs://hdfs:8020/flink/recovery//default/blob
2017-06-28 20:28:24,380 WARN org.apache.flink.configuration.Configuration
- Config uses deprecated configuration key
'recovery.zookeeper.quorum' instead of proper key
'high-availability.zookeeper.quorum'
2017-06-28 20:28:24,448 INFO org.apache.flink.runtime.util.ZooKeeperUtils
- Enforcing default ACL for ZK connections
2017-06-28 20:28:24,449 INFO org.apache.flink.runtime.util.ZooKeeperUtils
- Using '/flink/default' as Zookeeper namespace.
==>
/usr/local/flink/log/flink--taskmanager-0-flink-taskmanager-3923888361-c6fdw.out
<==
tail: unrecognized file system type 0x794c7630 for
‘/usr/local/flink/log/flink--taskmanager-0-flink-taskmanager-3923888361-c6fdw.log’.
please report this to bug-coreut...@gnu.org. reverting to polling
tail: unrecognized file system type 0x794c7630 for
‘/usr/local/flink/log/flink--taskmanager-0-flink-taskmanager-3923888361-c6fdw.out’.
please report this to bug-coreut...@gnu.org. reverting to polling
==>
/usr/local/flink/log/flink--taskmanager-0-flink-taskmanager-3923888361-c6fdw.log
<==
2017-06-28 20:28:24,564 INFO
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
- Starting
2017-06-28 20:28:24,569 INFO org.apache.zookeeper.ZooKeeper
- Client
environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f,
built on 03/23/2017 10:13 GMT
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:host.name=flink-taskmanager-3923888361-c6fdw
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.version=1.8.0_131
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.vendor=Oracle Corporation
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client
environment:java.class.path=/usr/local/flink-1.3.1/lib/egads-0.1.jar:/usr/local/flink-1.3.1/lib/flink-connector-kinesis_2.11-1.3.1.jar:/usr/local/flink-1.3.1/lib/flink-connector-rabbitmq_2.11-1.3.1.jar:/usr/local/flink-1.3.1/lib/flink-metrics-statsd-1.3.1.jar:/usr/local/flink-1.3.1/lib/flink-python_2.11-1.3.1.jar:/usr/local/flink-1.3.1/lib/flink-shaded-hadoop2-uber-1.3.1.jar:/usr/local/flink-1.3.1/lib/log4j-1.2.17.jar:/usr/local/flink-1.3.1/lib/slf4j-log4j12-1.7.7.jar:/usr/local/flink-1.3.1/lib/flink-dist_2.11-1.3.1.jar:::
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.io.tmpdir=/tmp
2017-06-28 20:28:24,570 INFO org.apache.zookeeper.ZooKeeper
- Client environment:java.compiler=<NA>
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.name=Linux
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.arch=amd64
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:os.version=4.11.6-coreos-r1
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.name=root
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.home=/root
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Client environment:user.dir=/
2017-06-28 20:28:24,571 INFO org.apache.zookeeper.ZooKeeper
- Initiating client connection, connectString=foo.bar:2181
sessionTimeout=60000
watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
2017-06-28 20:28:24,757 ERROR
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
- Background exception was not retry-able or retry gave up
java.net.UnknownHostException: foo.bar: No address associated with hostname
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at
org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.start(ConnectionState.java:109)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at
org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:128)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:96)
at
org.apache.flink.runtime.taskmanager.TaskManager$.selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1663)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1574)
at
org.apache.flink.runtime.taskmanager.TaskManager$$anon$2.call(TaskManager.scala:1572)
at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at
org.apache.flink.runtime.taskmanager.TaskManager$.main(TaskManager.scala:1572)
at
org.apache.flink.runtime.taskmanager.TaskManager.main(TaskManager.scala)
2017-06-28 20:28:24,850 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Using configured hostname/address for TaskManager: 10.2.45.10
2017-06-28 20:28:24,850 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Starting TaskManager
2017-06-28 20:28:24,850 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Starting TaskManager actor system at 10.2.45.10:6122.
2017-06-28 20:28:26,058 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2017-06-28 20:28:26,253 INFO Remoting
- Starting remoting
2017-06-28 20:28:26,848 INFO Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@10.2.45.10:6122]
2017-06-28 20:28:26,849 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Starting TaskManager actor
2017-06-28 20:28:26,863 INFO
org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig
[server address: /10.2.45.10, server port: 6121, ssl enabled: false, memory
segment size (bytes): 32768, transport type: NIO, number of server threads: 2
(manual), number of client threads: 2 (manual), server connect backlog: 0 (use
Netty's default), client connect timeout (sec): 120, send/receive buffer size
(bytes): 0 (use Netty's default)]
2017-06-28 20:28:26,953 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have
a max timeout of 10000 ms
2017-06-28 20:28:26,958 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory '/tmp': total 49 GB, usable 40 GB (81.63% usable)
2017-06-28 20:28:27,364 INFO
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 96 MB
for network buffer pool (number of memory segments: 3087, bytes per segment:
32768).
2017-06-28 20:28:27,859 INFO
org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the
network environment and its components.
2017-06-28 20:28:27,865 INFO
org.apache.flink.runtime.io.network.netty.NettyClient - Successful
initialization (took 2 ms).
2017-06-28 20:28:27,964 INFO
org.apache.flink.runtime.io.network.netty.NettyServer - Successful
initialization (took 99 ms). Listening on SocketAddress /10.2.45.10:6121.
2017-06-28 20:28:28,174 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting
managed memory to 0.7 of the currently free heap space (613 MB), memory will be
allocated lazily.
2017-06-28 20:28:28,250 INFO
org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager
uses directory /tmp/flink-io-df42316c-daf4-4791-9487-1ef9e28c3c94 for spill
files.
2017-06-28 20:28:28,255 INFO org.apache.flink.runtime.metrics.MetricRegistry
- Configuring StatsDReporter with {interval=10 SECONDS, port=8125,
host=localhost, class=org.apache.flink.metrics.statsd.StatsDReporter}.
2017-06-28 20:28:28,258 INFO org.apache.flink.metrics.statsd.StatsDReporter
- Configured StatsDReporter with {host:localhost, port:8125}
2017-06-28 20:28:28,258 INFO org.apache.flink.runtime.metrics.MetricRegistry
- Periodically reporting metrics in intervals of 10 SECONDS for
reporter statsd of type org.apache.flink.metrics.statsd.StatsDReporter.
2017-06-28 20:28:28,457 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/tmp/flink-dist-cache-a3f67484-1b8c-42ae-8c24-e442afac187e
2017-06-28 20:28:28,467 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/tmp/flink-dist-cache-1c26be7a-2132-4467-9fc3-827ec31a9e08
2017-06-28 20:28:28,551 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Starting TaskManager actor at
akka://flink/user/taskmanager#659221120.
2017-06-28 20:28:28,551 INFO org.apache.flink.runtime.taskmanager.TaskManager
- TaskManager data connection information:
3cd79e463ec19444d2024ac5700ccbae @ flink-taskmanager-3923888361-c6fdw
(dataPort=6121)
2017-06-28 20:28:28,551 INFO org.apache.flink.runtime.taskmanager.TaskManager
- TaskManager has 2 task slot(s).
2017-06-28 20:28:28,554 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Memory usage stats: [HEAP: 132/981/981 MB, NON HEAP: 43/44/-1 MB
(used/committed/max)]
2017-06-28 20:28:28,554 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
Starting ZooKeeperLeaderRetrievalService.
2017-06-28 20:28:43,568 ERROR
org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection
timed out for connection string (foo.bar:2181) and timeout (15000) / elapsed
(19000)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException:
KeeperErrorCode = ConnectionLoss
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at
org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:102)
at
org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-06-28 20:28:43,569 INFO org.apache.zookeeper.ZooKeeper
- Initiating client connection, connectString=foo.bar:2181
sessionTimeout=60000
watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
2017-06-28 20:29:03,578 ERROR
org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection
timed out for connection string (foo.bar:2181) and timeout (15000) / elapsed
(39013)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException:
KeeperErrorCode = ConnectionLoss
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at
org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:102)
at
org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-06-28 20:29:03,579 INFO org.apache.zookeeper.ZooKeeper
- Initiating client connection, connectString=foo.bar:2181
sessionTimeout=60000
watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
2017-06-28 20:29:33,727 WARN
org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection
attempt unsuccessful after 69161 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.
2017-06-28 20:29:33,727 INFO org.apache.zookeeper.ZooKeeper
- Initiating client connection, connectString=foo.bar:2181
sessionTimeout=60000
watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
2017-06-28 20:29:33,731 ERROR
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
- Ensure path threw exception
java.net.UnknownHostException: foo.bar: No address associated with hostname
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at
org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at
org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:102)
at
org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2017-06-28 20:29:33,748 WARN
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
Unhandled error encountered while retrieving Zookeeper leader.
2017-06-28 20:29:33,749 ERROR org.apache.flink.runtime.taskmanager.TaskManager
- Error in leader retrieval service
java.lang.RuntimeException: Ensure path threw exception
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.unhandledError(ZooKeeperLeaderRetrievalService.java:204)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:576)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:572)
at
org.apache.flink.shaded.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
at
org.apache.flink.shaded.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
org.apache.flink.shaded.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:571)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:100)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:102)
at
org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.UnknownHostException: foo.bar: No address associated with
hostname
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at
org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at
org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
... 20 more
2017-06-28 20:29:33,750 ERROR
org.apache.flink.shaded.org.apache.curator.framework.listen.ListenerContainer
- Listener
(org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService@32b7be44)
threw an exception
java.lang.RuntimeException: Ensure path threw exception
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.unhandledError(ZooKeeperLeaderRetrievalService.java:204)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:576)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$6.apply(CuratorFrameworkImpl.java:572)
at
org.apache.flink.shaded.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
at
org.apache.flink.shaded.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at
org.apache.flink.shaded.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.logError(CuratorFrameworkImpl.java:571)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:100)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
at
org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
at
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:102)
at
org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at
org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.UnknownHostException: foo.bar: No address associated with
hostname
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at
org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at
org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
at
org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at
org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at
org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at
org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
... 20 more
2017-06-28 20:29:33,763 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Stopping TaskManager akka://flink/user/taskmanager#659221120.
2017-06-28 20:29:33,763 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
Stopping ZooKeeperLeaderRetrievalService.
2017-06-28 20:29:33,768 INFO
org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager
removed spill file directory /tmp/flink-io-df42316c-daf4-4791-9487-1ef9e28c3c94
2017-06-28 20:29:33,769 INFO
org.apache.flink.runtime.io.network.NetworkEnvironment - Shutting down
the network environment and its components.
2017-06-28 20:29:33,774 INFO
org.apache.flink.runtime.io.network.netty.NettyClient - Successful
shutdown (took 4 ms).
2017-06-28 20:29:33,775 INFO
org.apache.flink.runtime.io.network.netty.NettyServer - Successful
shutdown (took 0 ms).
2017-06-28 20:29:33,864 INFO org.apache.flink.runtime.taskmanager.TaskManager
- Task manager akka://flink/user/taskmanager is completely shut
down.
2017-06-28 20:29:33,865 ERROR org.apache.flink.runtime.taskmanager.TaskManager
- Actor akka://flink/user/taskmanager#659221120 terminated,
stopping process...
{noformat}
> Flink Task Manager hangs on startup if one Zookeeper node is unresolvable
> -------------------------------------------------------------------------
>
> Key: FLINK-7021
> URL: https://issues.apache.org/jira/browse/FLINK-7021
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.3.1
> Environment: Kubernetes cluster running:
> * Flink 1.3.0 Job Manager & Task Manager on Java 8u131
> * Zookeeper 3.4.10 cluster with 3 nodes
> Reporter: Scott Kidder
>
> h2. Problem
> Flink Task Manager will hang during startup if one of the Zookeeper nodes in
> the Zookeeper connection string is unresolvable.
> h2. Expected Behavior
> Flink should retry name resolution & connection to Zookeeper nodes with
> exponential back-off.
> h2. Environment Details
> We're running Flink and Zookeeper in Kubernetes on CoreOS. CoreOS can run in
> a configuration that automatically detects and applies operating system
> updates. We have a Zookeeper node running on the same CoreOS instance as
> Flink. It's possible that the Zookeeper node will not yet be started when the
> Flink components are started. This could cause hostname resolution of the
> Zookeeper nodes to fail.
> h3. Flink Task Manager Logs
> {noformat}
> 2017-06-27 15:38:51,713 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Using
> configured hostname/address for TaskManager: 10.2.45.11
> 2017-06-27 15:38:51,714 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Starting
> TaskManager
> 2017-06-27 15:38:51,714 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Starting
> TaskManager actor system at 10.2.45.11:6122.
> 2017-06-27 15:38:52,950 INFO akka.event.slf4j.Slf4jLogger
> - Slf4jLogger started
> 2017-06-27 15:38:53,079 INFO Remoting
> - Starting remoting
> 2017-06-27 15:38:53,573 INFO Remoting
> - Remoting started; listening on addresses
> :[akka.tcp://flink@10.2.45.11:6122]
> 2017-06-27 15:38:53,576 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Starting
> TaskManager actor
> 2017-06-27 15:38:53,660 INFO
> org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig
> [server address: /10.2.45.11, server port: 6121, ssl enabled: false, memory
> segment size (bytes): 32768, transport type: NIO, number of server threads: 2
> (manual), number of client threads: 2 (manual), server connect backlog: 0
> (use Netty's default), client connect timeout (sec): 120, send/receive buffer
> size (bytes): 0 (use Netty's default)]
> 2017-06-27 15:38:53,682 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages
> have a max timeout of 10000 ms
> 2017-06-27 15:38:53,688 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary
> file directory '/tmp': total 49 GB, usable 42 GB (85.71% usable)
> 2017-06-27 15:38:54,071 INFO
> org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 96
> MB for network buffer pool (number of memory segments: 3095, bytes per
> segment: 32768).
> 2017-06-27 15:38:54,564 INFO
> org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the
> network environment and its components.
> 2017-06-27 15:38:54,576 INFO
> org.apache.flink.runtime.io.network.netty.NettyClient - Successful
> initialization (took 4 ms).
> 2017-06-27 15:38:54,677 INFO
> org.apache.flink.runtime.io.network.netty.NettyServer - Successful
> initialization (took 101 ms). Listening on SocketAddress /10.2.45.11:6121.
> 2017-06-27 15:38:54,981 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting
> managed memory to 0.7 of the currently free heap space (612 MB), memory will
> be allocated lazily.
> 2017-06-27 15:38:55,050 INFO
> org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager
> uses directory /tmp/flink-io-ca01554d-f25e-4c17-a828-96d82b43d4a7 for spill
> files.
> 2017-06-27 15:38:55,061 INFO org.apache.flink.runtime.metrics.MetricRegistry
> - Configuring StatsDReporter with {interval=10 SECONDS,
> port=8125, host=localhost,
> class=org.apache.flink.metrics.statsd.StatsDReporter}.
> 2017-06-27 15:38:55,065 INFO org.apache.flink.metrics.statsd.StatsDReporter
> - Configured StatsDReporter with {host:localhost, port:8125}
> 2017-06-27 15:38:55,065 INFO org.apache.flink.runtime.metrics.MetricRegistry
> - Periodically reporting metrics in intervals of 10 SECONDS for
> reporter statsd of type org.apache.flink.metrics.statsd.StatsDReporter.
> 2017-06-27 15:38:55,175 INFO org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /tmp/flink-dist-cache-e4c5bcc5-7513-40d9-a665-0d33c80a36ba
> 2017-06-27 15:38:55,187 INFO org.apache.flink.runtime.filecache.FileCache
> - User file cache uses directory
> /tmp/flink-dist-cache-310ba2f8-f96a-4c3f-b1db-35ac26b83f7e
> 2017-06-27 15:38:55,273 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Starting
> TaskManager actor at akka://flink/user/taskmanager#207081801.
> 2017-06-27 15:38:55,273 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - TaskManager
> data connection information: 7f86855dac2af4cca9eb2ae4c046630e @
> flink-taskmanager-3116622558-sqggc (dataPort=6121)
> 2017-06-27 15:38:55,273 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - TaskManager
> has 2 task slot(s).
> 2017-06-27 15:38:55,276 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Memory usage
> stats: [HEAP: 124/981/981 MB, NON HEAP: 43/44/-1 MB (used/committed/max)]
> 2017-06-27 15:38:55,276 INFO
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService -
> Starting ZooKeeperLeaderRetrievalService.
> 2017-06-27 15:39:10,289 ERROR
> org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection
> timed out for connection string
> (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181)
> and timeout (15000) / elapsed (18617)
> org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException:
> KeeperErrorCode = ConnectionLoss
> at
> org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
> at
> org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
> at
> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
> at
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:100)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
> at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
> at akka.actor.ActorCell.create(ActorCell.scala:580)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2017-06-27 15:39:30,349 INFO org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
> sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
> 2017-06-27 15:40:00,388 WARN
> org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection
> attempt unsuccessful after 68719 (greater than max timeout of 60000).
> Resetting connection and trying again with a new connection.
> 2017-06-27 15:40:00,388 INFO org.apache.zookeeper.ZooKeeper
> - Initiating client connection,
> connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
> sessionTimeout=60000
> watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@16f7b4af
> 2017-06-27 15:40:00,450 ERROR
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
> - Ensure path threw exception
> java.net.UnknownHostException: zookeeper-1.zookeeper: Name or service not
> known
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
> at
> org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
> at
> org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
> at
> org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
> at
> org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
> at
> org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
> at
> org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
> at
> org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
> at
> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
> at
> org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158)
> at
> org.apache.flink.shaded.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.reset(NodeCache.java:242)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:175)
> at
> org.apache.flink.shaded.org.apache.curator.framework.recipes.cache.NodeCache.start(NodeCache.java:154)
> at
> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService.start(ZooKeeperLeaderRetrievalService.java:100)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.preStart(TaskManager.scala:205)
> at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.aroundPreStart(TaskManager.scala:120)
> at akka.actor.ActorCell.create(ActorCell.scala:580)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
