2018-07-03 11:30:35,703 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-07-03 11:30:35,705 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting StandaloneSessionClusterEntrypoint (Version: <unknown>,
Rev:c61b108, Date:24.05.2018 @ 16:54:44 CEST)
2018-07-03 11:30:35,705 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS
current user: flink
2018-07-03 11:30:35,705 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM:
OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Maximum heap size: 981 MiBytes
2018-07-03 11:30:35,706 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
JAVA_HOME: /etc/alternatives/jre_openjdk/
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No
Hadoop Dependency available
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM
Options:
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xms1024m
2018-07-03 11:30:35,707 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xmx1024m
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:/opt/flink-1.5.0/conf/log4j.properties
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlogback.configurationFile=file:/opt/flink-1.5.0/conf/logback.xml
2018-07-03 11:30:35,708 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Program Arguments:
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--configDir
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
/opt/flink-1.5.0/conf
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--executionMode
2018-07-03 11:30:35,709 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
cluster
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--host
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
cluster
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Classpath:
/opt/flink-1.5.0/lib/flink-cep_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-connectors-1.5.0.jar:/opt/flink-1.5.0/lib/flink-gelly_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-ml_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-table_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/log4j-1.2.17.jar:/opt/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/opt/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar:::
2018-07-03 11:30:35,710 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-07-03 11:30:35,712 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Registered UNIX signal handlers for [TERM, HUP, INT]
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: blob.server.port, 4124
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.rpc.address,
myfl-flink-jobmanager
2018-07-03 11:30:35,720 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.rpc.port, 4123
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.heap.mb, 1024
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.heap.mb, 1024
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.rpc.port, 4122
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.data.port, 4121
2018-07-03 11:30:35,721 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.query.port, 4125
2018-07-03 11:30:35,722 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: rest.port, 8081
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: rest.address, myfl-flink-jobmanager-ui
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: state.backend, rocksdb
2018-07-03 11:30:35,762 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: state.checkpoints.dir,
file:///opt/flink/share/myfl-flink/checkpoints/ext_checkpoints
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: state.backend.fs.checkpointdir,
file:///opt/flink/share/myfl-flink/checkpoints/fs_state
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: state.backend.rocksdb.checkpointdir,
file:///opt/flink/share/myfl-flink/checkpoints/rocksdb_state
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.numberOfTaskSlots, 4
2018-07-03 11:30:35,763 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: restart-strategy, fixed-delay
2018-07-03 11:30:35,764 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: restart-strategy.fixed-delay.attempts,
100
2018-07-03 11:30:35,764 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: restart-strategy.fixed-delay.delay, 1
s
2018-07-03 11:30:35,885 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting StandaloneSessionClusterEntrypoint.
2018-07-03 11:30:35,885 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Install default filesystem.
2018-07-03 11:30:35,892 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies.
The extended set of supported File Systems via Hadoop is not
available.
2018-07-03 11:30:35,963 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Install security context.
2018-07-03 11:30:35,970 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory -
Cannot create Hadoop Security Module because Hadoop cannot be found in
the Classpath.
2018-07-03 11:30:35,988 INFO
org.apache.flink.runtime.security.SecurityUtils - Cannot
install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2018-07-03 11:30:35,989 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Initializing cluster services.
2018-07-03 11:30:36,003 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying
to start actor system at myfl-flink-jobmanager:4123
2018-07-03 11:30:37,288 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2018-07-03 11:30:37,396 INFO akka.remote.Remoting
- Starting remoting
2018-07-03 11:30:37,583 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@myfl-flink-jobmanager:4123]
2018-07-03 11:30:37,591 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor
system started at akka.tcp://flink@myfl-flink-jobmanager:4123
2018-07-03 11:30:37,611 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/tmp/blobStore-e445bc66-cee3-4a3d-b810-74df02627eca
2018-07-03 11:30:37,613 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at 0.0.0.0:4124 - max
concurrent requests: 50 - max backlog: 1000
2018-07-03 11:30:37,629 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No
metrics reporter configured, no metrics will be exposed/reported.
2018-07-03 11:30:37,664 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore -
Initializing FileArchivedExecutionGraphStore: Storage directory
/tmp/executionGraphStore-4ff546b1-4bfb-4911-9314-89c61d7e7149,
expiration time 3600000, maximum cache size 52428800 bytes.
2018-07-03 11:30:37,694 INFO
org.apache.flink.runtime.blob.TransientBlobCache -
Created BLOB cache storage directory
/tmp/blobStore-7e0efdb8-f70b-42ed-9387-c0e1b8090b36
2018-07-03 11:30:37,702 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload
directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded
files are no longer available.
2018-07-03 11:30:37,703 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Created directory
/tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
for file uploads.
2018-07-03 11:30:37,706 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Starting rest endpoint.
2018-07-03 11:30:38,369 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component log file:
/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log
2018-07-03 11:30:38,369 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component stdout file:
/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.out
2018-07-03 11:30:38,567 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest
endpoint listening at myfl-flink-jobmanager-ui:8081
2018-07-03 11:30:38,568 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
http://myfl-flink-jobmanager-ui:8081 was granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2018-07-03 11:30:38,568 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web
frontend listening at http://myfl-flink-jobmanager-ui:8081.
2018-07-03 11:30:38,578 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at
akka://flink/user/resourcemanager .
2018-07-03 11:30:38,966 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .
2018-07-03 11:30:39,068 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
ResourceManager
akka.tcp://flink@myfl-flink-jobmanager:4123/user/resourcemanager was
granted leadership with fencing token 00000000000000000000000000000000
2018-07-03 11:30:39,069 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
Starting the SlotManager.
2018-07-03 11:30:39,164 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Dispatcher akka.tcp://flink@myfl-flink-jobmanager:4123/user/dispatcher
was granted leadership with fencing token
00000000000000000000000000000000
2018-07-03 11:30:39,165 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Recovering all persisted jobs.
2018-07-03 11:30:39,682 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
Replacing old registration of TaskExecutor
068c693b9585900f68c53b00507ee889.
2018-07-03 11:30:39,683 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
Unregister TaskManager 8a5cee3aa38081030dc8558ac477d3b3 from the
SlotManager.
2018-07-03 11:30:39,683 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
The target with resource ID 068c693b9585900f68c53b00507ee889 is
already been monitored.
2018-07-03 11:30:39,770 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
Registering TaskManager 068c693b9585900f68c53b00507ee889 under
03d409e5166fad4f4082b6165eb0de2e at the SlotManager.
2018-07-03 11:34:20,257 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Submitting job b684656d9afd75cc384a7bcd071bf55e (CSV Files Read -> CSV
to Avro encode -> Kafka publish).
2018-07-03 11:34:20,269 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster
at akka://flink/user/jobmanager_0 .
2018-07-03 11:34:20,278 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Initializing job CSV Files Read -> CSV to Avro encode -> Kafka publish
(b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,285 INFO
org.apache.flink.runtime.jobmaster.JobMaster - Using
restart strategy
FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647,
delayBetweenRestartAttempts=0) for CSV Files Read -> CSV to Avro
encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,289 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.slotpool.SlotPool at
akka://flink/user/67ceb2ae-1cb1-44be-a09e-601032e23fb5 .
2018-07-03 11:34:20,481 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
recovers via failover strategy: full graph restart
2018-07-03 11:34:20,562 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Running initialization on master for job CSV Files Read -> CSV to Avro
encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e).
2018-07-03 11:34:20,563 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Successfully ran initialization on master in 0 ms.
2018-07-03 11:34:20,580 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Loading state backend via factory
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2018-07-03 11:34:20,590 WARN
org.apache.flink.configuration.Configuration - Config
uses deprecated configuration key
'state.backend.rocksdb.checkpointdir' instead of proper key
'state.backend.rocksdb.localdir'
2018-07-03 11:34:20,592 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler -
Implementation error: Unhandled exception.
org.apache.flink.util.FlinkException: Failed to submit job
b684656d9afd75cc384a7bcd071bf55e.
at
org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.client.JobExecutionException:
Could not set up JobManager
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169)
at
org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885)
at
org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287)
at
org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277)
at
org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:262)
at
org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:249)
... 21 more
Caused by: org.apache.flink.runtime.client.JobExecutionException:
Could not instantiate configured state backend
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
at
org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1150)
at
org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1130)
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:298)
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:151)
... 26 more
Caused by: org.apache.flink.configuration.IllegalConfigurationException:
Invalid configuration for RocksDB state backend's local storage
directories: Relative paths are not supported
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:273)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:296)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32)
at
org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:157)
at
org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222)
at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304)
... 31 more
Caused by: java.lang.IllegalArgumentException: Relative paths are not supported
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.setDbStoragePaths(RocksDBStateBackend.java:518)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:269)
... 37 more
On Tue, Jul 3, 2018 at 5:11 PM, Chesnay Schepler <ches...@apache.org> wrote:
> Doesn't sound like intended behavior, can you give us the stacktrace?
>
>
> On 03.07.2018 13:17, Data Engineer wrote:
>
>> The Flink documentation says that we need to specify the filesystem type
>> (file://, hdfs://) when configuring the rocksdb backend dir.
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/
>> ops/state/state_backends.html#the-rocksdbstatebackend
>>
>> But when I do this, I get an error on job submission saying that relative
>> paths are not permitted in the rocksdb stand backend.
>> I am submitting the job via flink cli (bin/flink run).
>>
>> Also, even though I give a local file system path
>> "file:///home/abc/share", it is a shared GlusterFS volume mount, so it will
>> be accessible by the JobManager and all TaskManagers.
>>
>> I removed the filesystem type from the rocksdb backend dir configuration,
>> and though the job got submitted, the rocksdb checkpoint directory was not
>> created.
>> I have enabled checkpointing in my Flink application.
>>
>> I am using Flink 1.5.0.
>>
>> Any help or pointers would be appreciated.
>>
>
>
>
