2018-07-03 11:30:35,703 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-07-03 11:30:35,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint
- Starting StandaloneSessionClusterEntrypoint (Version: <unknown>,
Rev:c61b108, Date:24.05.2018 @ 16:54:44 CEST)
2018-07-03 11:30:35,705 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current
user: flink
2018-07-03 11:30:35,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint
- Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK
64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap
size: 981 MiBytes
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME:
/etc/alternatives/jre_openjdk/
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop
Dependency available
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:/opt/flink-1.5.0/conf/log4j.properties
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlogback.configurationFile=file:/opt/flink-1.5.0/conf/logback.xml
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program
Arguments:
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
/opt/flink-1.5.0/conf
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--executionMode
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath:
/opt/flink-1.5.0/lib/flink-cep_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-connectors-1.5.0.jar:/opt/flink-1.5.0/lib/flink-gelly_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-ml_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-table_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/log4j-1.2.17.jar:/opt/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/opt/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar:::
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-07-03 11:30:35,712 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX
signal handlers for [TERM, HUP, INT]
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: blob.server.port, 4124
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, myfl-flink-jobmanager
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 4123
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.mb, 1024
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.mb, 1024
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.rpc.port, 4122
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.data.port, 4121
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.query.port, 4125
2018-07-03 11:30:35,722 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.port, 8081
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.address, myfl-flink-jobmanager-ui
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.backend, rocksdb
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.checkpoints.dir,
file:///opt/flink/share/myfl-flink/checkpoints/ext_checkpoints
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.backend.fs.checkpointdir,
file:///opt/flink/share/myfl-flink/checkpoints/fs_state
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.backend.rocksdb.checkpointdir,
file:///opt/flink/share/myfl-flink/checkpoints/rocksdb_state
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 4
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: restart-strategy, fixed-delay
2018-07-03 11:30:35,764 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: restart-strategy.fixed-delay.attempts, 100
2018-07-03 11:30:35,764 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: restart-strategy.fixed-delay.delay, 1 s
2018-07-03 11:30:35,885 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting
StandaloneSessionClusterEntrypoint.
2018-07-03 11:30:35,885 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default
filesystem.
2018-07-03 11:30:35,892 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies. The extended set of
supported File Systems via Hadoop is not available.
2018-07-03 11:30:35,963 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install
security context.
2018-07-03 11:30:35,970 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create
Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-07-03 11:30:35,988 INFO org.apache.flink.runtime.security.SecurityUtils
- Cannot install HadoopSecurityContext because Hadoop cannot be
found in the Classpath.
2018-07-03 11:30:35,989 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing
cluster services.
2018-07-03 11:30:36,003 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start
actor system at myfl-flink-jobmanager:4123
2018-07-03 11:30:37,288 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2018-07-03 11:30:37,396 INFO akka.remote.Remoting
- Starting remoting
2018-07-03 11:30:37,583 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@myfl-flink-jobmanager:4123]
2018-07-03 11:30:37,591 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system
started at akka.tcp://flink@myfl-flink-jobmanager:4123
2018-07-03 11:30:37,611 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/tmp/blobStore-e445bc66-cee3-4a3d-b810-74df02627eca
2018-07-03 11:30:37,613 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at0.0.0.0:4124 <http://0.0.0.0:4124> - max
concurrent requests: 50 - max backlog: 1000
2018-07-03 11:30:37,629 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.
2018-07-03 11:30:37,664 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore -
Initializing FileArchivedExecutionGraphStore: Storage directory
/tmp/executionGraphStore-4ff546b1-4bfb-4911-9314-89c61d7e7149, expiration time
3600000, maximum cache size 52428800 bytes.
2018-07-03 11:30:37,694 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-7e0efdb8-f70b-42ed-9387-c0e1b8090b36
2018-07-03 11:30:37,702 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload
directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded files are
no longer available.
2018-07-03 11:30:37,703 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created
directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
for file uploads.
2018-07-03 11:30:37,706 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest
endpoint.
2018-07-03 11:30:38,369 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined
location of main cluster component log file:
/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log
2018-07-03 11:30:38,369 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined
location of main cluster component stdout file:
/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.out
2018-07-03 11:30:38,567 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint
listening at myfl-flink-jobmanager-ui:8081
2018-07-03 11:30:38,568 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint
-http://myfl-flink-jobmanager-ui:8081 was granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2018-07-03 11:30:38,568 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend
listening athttp://myfl-flink-jobmanager-ui:8081.
2018-07-03 11:30:38,578 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at
akka://flink/user/resourcemanager .
2018-07-03 11:30:38,966 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .
2018-07-03 11:30:39,068 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
ResourceManager
akka.tcp://flink@myfl-flink-jobmanager:4123/user/resourcemanager was granted
leadership with fencing token 00000000000000000000000000000000
2018-07-03 11:30:39,069 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting
the SlotManager.
2018-07-03 11:30:39,164 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher
akka.tcp://flink@myfl-flink-jobmanager:4123/user/dispatcher was granted
leadership with fencing token 00000000000000000000000000000000
2018-07-03 11:30:39,165 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all
persisted jobs.
2018-07-03 11:30:39,682 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Replacing
old registration of TaskExecutor 068c693b9585900f68c53b00507ee889.
2018-07-03 11:30:39,683 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Unregister
TaskManager 8a5cee3aa38081030dc8558ac477d3b3 from the SlotManager.
2018-07-03 11:30:39,683 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - The
target with resource ID 068c693b9585900f68c53b00507ee889 is already been
monitored.
2018-07-03 11:30:39,770 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering
TaskManager 068c693b9585900f68c53b00507ee889 under
03d409e5166fad4f4082b6165eb0de2e at the SlotManager.
2018-07-03 11:34:20,257 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job
b684656d9afd75cc384a7bcd071bf55e (CSV Files Read -> CSV to Avro encode -> Kafka
publish).
2018-07-03 11:34:20,269 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
2018-07-03 11:34:20,278 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Initializing job CSV Files Read -> CSV to Avro encode -> Kafka publish
(b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,285 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Using restart strategy
FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647,
delayBetweenRestartAttempts=0) for CSV Files Read -> CSV to Avro encode ->
Kafka publish (b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,289 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.slotpool.SlotPool at
akka://flink/user/67ceb2ae-1cb1-44be-a09e-601032e23fb5 .
2018-07-03 11:34:20,481 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers
via failover strategy: full graph restart
2018-07-03 11:34:20,562 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Running initialization on master for job CSV Files Read -> CSV to Avro
encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,563 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Successfully ran initialization on master in 0 ms.
2018-07-03 11:34:20,580 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Loading state backend via factory
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2018-07-03 11:34:20,590 WARN org.apache.flink.configuration.Configuration
- Config uses deprecated configuration key
'state.backend.rocksdb.checkpointdir' instead of proper key
'state.backend.rocksdb.localdir'
2018-07-03 11:34:20,592 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation
error: Unhandled exception.
org.apache.flink.util.FlinkException: Failed to submit job
b684656d9afd75cc384a7bcd071bf55e.
at
org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set
up JobManager
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169)
at
org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885)
at
org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287)
at
org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277)
at
org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:262)
at
org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:249)
... 21 more
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not
instantiate configured state backend
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
at
org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1150)
at
org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1130)
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:298)
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:151)
... 26 more
Caused by: org.apache.flink.configuration.IllegalConfigurationException:
Invalid configuration for RocksDB state backend's local storage directories:
Relative paths are not supported
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:273)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:296)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32)
at
org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:157)
at
org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304)
... 31 more
Caused by: java.lang.IllegalArgumentException: Relative paths are not supported
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.setDbStoragePaths(RocksDBStateBackend.java:518)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:269)
... 37 more
On Tue, Jul 3, 2018 at 5:11 PM, Chesnay Schepler <ches...@apache.org
<mailto:ches...@apache.org>> wrote:
Doesn't sound like intended behavior, can you give us the stacktrace?
On 03.07.2018 13:17, Data Engineer wrote:
The Flink documentation says that we need to specify the
filesystem type (file://, hdfs://) when configuring the
rocksdb backend dir.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html#the-rocksdbstatebackend
<https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html#the-rocksdbstatebackend>
But when I do this, I get an error on job submission saying
that relative paths are not permitted in the rocksdb stand
backend.
I am submitting the job via flink cli (bin/flink run).
Also, even though I give a local file system path
"file:///home/abc/share", it is a shared GlusterFS volume
mount, so it will be accessible by the JobManager and all
TaskManagers.
I removed the filesystem type from the rocksdb backend dir
configuration, and though the job got submitted, the rocksdb
checkpoint directory was not created.
I have enabled checkpointing in my Flink application.
I am using Flink 1.5.0.
Any help or pointers would be appreciated.
