On Tue, Dec 19, 2017 at 7:29 PM, Colin Williams < colin.williams.seat...@gmail.com> wrote:
> Hi, > > I've been trying to update my flink-docker jobmanager configuration for > flink 1.4. I think the system is shutting down after a leadership election, > but I'm not sure what the issue is. My configuration of the jobmanager > follows > > > jobmanager.rpc.address: 10.16.228.150 > jobmanager.rpc.port: 6123 > jobmanager.heap.mb: 1024 > blob.server.port: 6124 > query.server.port: 6125 > > web.port: 8081 > web.history: 10 > > parallelism.default: 1 > > state.backend: rocksdb > state.backend.rocksdb.checkpointdir: /tmp/flink/rocksdb > state.backend.fs.checkpointdir: file:///var/lib/data/checkpoints > > high-availability: zookeeper > high-availability.cluster-id: /dev > high-availability.zookeeper.quorum: 10.16.228.190:2181 > high-availability.zookeeper.path.root: /flink-1.4 > high-availability.zookeeper.storageDir: file:///var/lib/data/recovery > high-availability.jobmanager.port: 50010 > > env.java.opts: -Dlog.file=/opt/flink/log/jobmanager.log > > I'm also attaching some debugging output which shows the shutdown. Again > I'm not entirely sure it's caused by a leadership issue because it's not > clear from the debug logs. Can anyone suggest changes I might make to the > configuration to fix this? I've tried clearing the zookeeper root path in > case it had some old session information, but that didn't seem to help. > > Best, > > Colin Williams >
DEBUG org.apache.flink.shaded.akka.org.jboss.netty.channel.socket.nio.SelectorUtil - Using select timeout of 500 DEBUG org.apache.flink.shaded.akka.org.jboss.netty.channel.socket.nio.SelectorUtil - Epoll-bug workaround enabled = false INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@10.16.228.150:50010] INFO org.apache.flink.runtime.jobmanager.JobManager - Actor system started at akka.tcp://flink@10.16.228.150:50010 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'high-availability.zookeeper.storageDir' instead of proper key 'high-availability.storageDir' INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at file:///var/lib/data/recovery//dev/blob DEBUG org.apache.flink.runtime.blob.FileSystemBlobStore - Created highly available BLOB storage directory at file:///var/lib/data/recovery//dev/blob INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-1.4/dev' as Zookeeper namespace. INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting DEBUG org.apache.flink.shaded.curator.org.apache.curator.CuratorZookeeperClient - Starting DEBUG org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Starting DEBUG org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - reset INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=ip-10-16-228-150.us-east-2.compute.internal INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_151 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/flink/lib/flink-python_2.11-1.4.0.jar:/opt/flink/lib/flink-shaded-hadoop2-uber-1.4.0.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.4.0.jar::: INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.13.9-coreos INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/flink INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.16.228.190:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@6f8f9349 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-838316936299798682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.16.228.190/10.16.228.190:2181 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to 10.16.228.190/10.16.228.190:2181, initiating session DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment request sent on 10.16.228.190/10.16.228.190:2181 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.16.228.190/10.16.228.190:2181, sessionid = 0x15f473a91e610a4, negotiated timeout = 40000 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager web frontend INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /opt/flink/log/jobmanager.log INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /opt/flink/log/jobmanager.out INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-10ef62de-8458-4843-94c2-9889fcf147f1 for the web interface files INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-647e10b5-58b1-4092-966e-ec74934fdb68 for web frontend JAR file uploads DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - Java version: 8 DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.noJavassist: false DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - Javassist: available DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model) DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.shaded.netty4.io.netty.handler.codec.http.router.RoutedMatcher DEBUG org.apache.flink.shaded.netty4.io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 4 DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false DEBUG org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512 DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x47d1a6a7ced7be7e (took 0 ms) DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled DEBUG org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536 DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) DEBUG org.apache.flink.shaded.netty4.io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 2048 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 10.16.228.150:8081 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor DEBUG org.apache.flink.runtime.jobmanager.JobManager - No archive directory was configured. Jobs will not be archived. INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-37b11802-9515-41bb-a150-5269c5b87bf3 DEBUG org.apache.flink.util.NetUtils - Trying to open socket on port 6124 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'high-availability.zookeeper.storageDir' instead of proper key 'high-availability.storageDir' INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 1,3 replyHeader:: 1,135069,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 2,3 replyHeader:: 2,135069,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 3,3 replyHeader:: 3,135069,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 4,3 replyHeader:: 4,135069,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 5,3 replyHeader:: 5,135069,0 request:: '/flink-1.4/dev/jobgraphs,F response:: s{135032,135032,1513738651056,1513738651056,0,0,0,0,0,0,135032} DEBUG org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager process reaper INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink@10.16.228.150:50010/user/jobmanager. INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService@40d08669. INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService. DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 6,3 replyHeader:: 6,135069,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 7,3 replyHeader:: 7,135069,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 8,3 replyHeader:: 8,135069,0 request:: '/flink-1.4/dev/leader,F response:: s{135033,135033,1513738651102,1513738651102,0,1,0,0,0,1,135034} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 9,3 replyHeader:: 9,135069,0 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000,F response:: s{135034,135034,1513738651107,1513738651107,0,2,0,0,0,0,135047} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock serverPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock finished:false header:: 10,3 replyHeader:: 10,135069,-101 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock,T response:: DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 11,3 replyHeader:: 11,135069,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 12,3 replyHeader:: 12,135069,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.runtime.jobmanager.JobManager - Starting Resource manager actor INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService. DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 13,3 replyHeader:: 13,135069,0 request:: '/flink-1.4/dev/leader,F response:: s{135033,135033,1513738651102,1513738651102,0,1,0,0,0,1,135034} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 14,3 replyHeader:: 14,135069,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 15,3 replyHeader:: 15,135069,0 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000,F response:: s{135034,135034,1513738651107,1513738651107,0,2,0,0,0,0,135047} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_e7fc3dcd-c1ec-42ff-83ff-b9dacb80f367-latch- serverPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_e7fc3dcd-c1ec-42ff-83ff-b9dacb80f367-latch- finished:false header:: 16,1 replyHeader:: 16,135070,0 request:: '/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_e7fc3dcd-c1ec-42ff-83ff-b9dacb80f367-latch-,,v{s{31,s{'world,'anyone}}},3 response:: '/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_e7fc3dcd-c1ec-42ff-83ff-b9dacb80f367-latch-0000000011 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 17,3 replyHeader:: 17,135070,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock serverPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock finished:false header:: 18,3 replyHeader:: 18,135070,-101 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock,T response:: DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 19,3 replyHeader:: 19,135070,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock serverPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock finished:false header:: 20,12 replyHeader:: 20,135070,0 request:: '/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock,F response:: v{'_c_f4ae53e8-812d-40d9-8d52-53761c34e66f-latch-0000000008,'_c_e7fc3dcd-c1ec-42ff-83ff-b9dacb80f367-latch-0000000011,'_c_486066d1-d242-4eb6-a8dd-0da5f6fa5f3d-latch-0000000009,'_c_cf66c8b0-9c34-45fb-8acc-78c7282a5122-latch-0000000010},s{135038,135038,1513738651859,1513738651859,0,20,0,0,0,4,135070} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 21,3 replyHeader:: 21,135070,0 request:: '/flink-1.4/dev/jobgraphs,F response:: s{135032,135032,1513738651056,1513738651056,0,0,0,0,0,0,135032} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 22,3 replyHeader:: 22,135070,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 23,3 replyHeader:: 23,135070,0 request:: '/flink-1.4,F response:: s{135030,135030,1513738651033,1513738651033,0,1,0,0,0,1,135031} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 24,3 replyHeader:: 24,135070,0 request:: '/flink-1.4/dev/leader,F response:: s{135033,135033,1513738651102,1513738651102,0,1,0,0,0,1,135034} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_cf66c8b0-9c34-45fb-8acc-78c7282a5122-latch-0000000010 serverPath:/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_cf66c8b0-9c34-45fb-8acc-78c7282a5122-latch-0000000010 finished:false header:: 25,4 replyHeader:: 25,135070,0 request:: '/flink-1.4/dev/leaderlatch/00000000000000000000000000000000/job_manager_lock/_c_cf66c8b0-9c34-45fb-8acc-78c7282a5122-latch-0000000010,T response:: ,s{135067,135067,1513738760996,1513738760996,0,0,0,98876033706889379,0,0,135067} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 26,3 replyHeader:: 26,135070,0 request:: '/flink-1.4/dev,F response:: s{135031,135031,1513738651043,1513738651043,0,3,0,0,0,3,135036} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 27,3 replyHeader:: 27,135070,0 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000,F response:: s{135034,135034,1513738651107,1513738651107,0,2,0,0,0,0,135047} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 28,3 replyHeader:: 28,135070,0 request:: '/flink-1.4/dev/jobgraphs,F response:: s{135032,135032,1513738651056,1513738651056,0,0,0,0,0,0,135032} DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock serverPath:/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock finished:false header:: 29,3 replyHeader:: 29,135070,-101 request:: '/flink-1.4/dev/leader/00000000000000000000000000000000/job_manager_lock,T response:: DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:null serverPath:null finished:false header:: 30,3 replyHeader:: 30,135070,-101 request:: '/flink-1.4/dev/jobgraphs/foo,F response:: DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x15f473a91e610a4, packet:: clientPath:/flink-1.4/dev/jobgraphs serverPath:/flink-1.4/dev/jobgraphs finished:false header:: 31,12 replyHeader:: 31,135070,0 request:: '/flink-1.4/dev/jobgraphs,T response:: v{},s{135032,135032,1513738651056,1513738651056,0,0,0,0,0,0,135032} DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpRequestMatcher DEBUG org.apache.flink.shaded.netty4.io.netty.util.ResourceLeakDetector - -Dorg.apache.flink.shaded.netty4.io.netty.leakDetectionLevel: simple DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: org.apache.flink.shaded.netty4.io.netty.util.internal.__matchers__.org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectMatcher DEBUG org.apache.flink.shaded.netty4.io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144 DEBUG org.apache.flink.shaded.netty4.io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available ce... INFO org.apache.flink.runtime.jobmanager.JobManager - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested. INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard root cache directory /tmp/flink-web-10ef62de-8458-4843-94c2-9889fcf147f1 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard jar upload directory /tmp/flink-web-647e10b5-58b1-4092-966e-ec74934fdb68 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:6124