Hi Ufuk,
sorry for not getting back to you for so long, and thanks for your
answer. The problem persists unfortunately. Running the job from the IDE
works (with core-site.xml on classpath), running it in local standalone
mode does not. AccessKeyID and SecretAccesKey are not found.
Attached the jobmanager log on DEBUG level. The core-site.xml is
definitely at the configured location.
I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to
run the jar in local mode. Do I have to use the Hadoop 2.x version for
this to work? I have put hadoop-common-2.3.jar into the flink lib folder.
I don't know if it is relevant (but it seems to be related), when I run
the job from my IDE I get the warning:
2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Cheers and thank you,
Konstantin
On 14.10.2015 11:44, Ufuk Celebi wrote:
>
>> On 10 Oct 2015, at 22:59, snntr <konstantin.kn...@tngtech.com> wrote:
>>
>> Hey everyone,
>>
>> I was having the same problem with S3 and found this thread very useful.
>> Everything works fine now, when I start Flink from my IDE, but when I run
>> the jar in local mode I keep getting
>>
>> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
>> must be specified as the username or password (respectively) of a s3n URL,
>> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
>> properties (respectively).
>>
>> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
>> machine with the required properties. What am I missing?
>>
>> Any advice is highly appreciated ;)
>
> This looks like a problem with picking up the Hadoop config. Can you look
> into the logs to check whether the configuration is picked up? Change the log
> settings to DEBUG in log/log4j.properties for this. And can you provide the
> complete stack trace?
>
> – Ufuk
>
>
--
Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082
12:29:36,538 DEBUG org.apache.hadoop.security.Groups - Creating new Groups object
12:29:36,559 DEBUG org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
12:29:36,590 DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login
12:29:36,591 DEBUG org.apache.hadoop.security.UserGroupInformation - hadoop login commit
12:29:36,593 DEBUG org.apache.hadoop.security.UserGroupInformation - using local user:UnixPrincipal: kknauf
12:29:36,594 DEBUG org.apache.hadoop.security.UserGroupInformation - UGI loginUser:kknauf
12:29:36,594 INFO org.apache.flink.runtime.jobmanager.JobManager - --------------------------------------------------------------------------------
12:29:36,594 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager (Version: 0.10.0, Rev:ab2cca4, Date:10.11.2015 @ 13:50:14 UTC)
12:29:36,595 INFO org.apache.flink.runtime.jobmanager.JobManager - Current user: kknauf
12:29:36,595 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.66-b17
12:29:36,595 INFO org.apache.flink.runtime.jobmanager.JobManager - Maximum heap size: 736 MiBytes
12:29:36,595 INFO org.apache.flink.runtime.jobmanager.JobManager - JAVA_HOME: /usr/lib/jvm/java-8-oracle/
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - Hadoop version: 1.2.1
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM Options:
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xms768m
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xmx768m
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog.file=/opt/flink-0.10.0/log/flink-kknauf-jobmanager-0-kknauf-ThinkPad-T440p.log
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog4j.configuration=file:/opt/flink-0.10.0/conf/log4j.properties
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlogback.configurationFile=file:/opt/flink-0.10.0/conf/logback.xml
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - Program Arguments:
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - --configDir
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - /opt/flink-0.10.0/conf
12:29:36,599 INFO org.apache.flink.runtime.jobmanager.JobManager - --executionMode
12:29:36,600 INFO org.apache.flink.runtime.jobmanager.JobManager - local
12:29:36,600 INFO org.apache.flink.runtime.jobmanager.JobManager - --streamingMode
12:29:36,600 INFO org.apache.flink.runtime.jobmanager.JobManager - batch
12:29:36,600 INFO org.apache.flink.runtime.jobmanager.JobManager - --------------------------------------------------------------------------------
12:29:36,693 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /opt/flink-0.10.0/conf
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 256
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 512
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
12:29:36,699 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
12:29:36,700 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
12:29:36,700 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: webclient.port, 8080
12:29:36,700 DEBUG org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /home/kknauf/core-site.xml
12:29:36,702 INFO org.apache.flink.runtime.jobmanager.JobManager - Staring JobManager without high-availability
12:29:36,706 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on 127.0.0.1:6123 with execution mode LOCAL and streaming mode BATCH_ONLY
12:29:36,715 INFO org.apache.flink.runtime.jobmanager.JobManager - Security is not enabled. Starting non-authenticated JobManager.
12:29:36,716 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager
12:29:36,716 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system at 127.0.0.1:6123
12:29:36,763 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Using akka configuration
Config(SimpleConfigObject({"akka":{"actor":{"default-dispatcher":{"fork-join-executor":{"parallelism-factor":2},"throughput":15},"guardian-supervisor-strategy":"akka.actor.StoppingSupervisorStrategy","provider":"akka.remote.RemoteActorRefProvider"},"daemonic":"on","jvm-exit-on-fatal-error":"off","log-config-on-start":"off","log-dead-letters":"off","log-dead-letters-during-shutdown":"off","loggers":["akka.event.slf4j.Slf4jLogger"],"logging-filter":"akka.event.slf4j.Slf4jLoggingFilter","loglevel":"DEBUG","remote":{"log-remote-lifecycle-events":"off","netty":{"tcp":{"connection-timeout":"100 seconds","hostname":"127.0.0.1","maximum-frame-size":"10485760b","port":6123,"tcp-nodelay":"on","transport-class":"akka.remote.transport.netty.NettyTransport"}},"startup-timeout":"100 seconds","transport-failure-detector":{"acceptable-heartbeat-pause":"6000 s","heartbeat-interval":"1000 s","threshold":300},"watch-failure-detector":{"acceptable-heartbeat-pause":"100 seconds","heartbeat-interval":"10 seconds","threshold":12}},"serialize-messages":"off","stdout-loglevel":"OFF"}}))
12:29:36,909 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
12:29:36,917 DEBUG akka.event.EventStream - logger log1-Slf4jLogger started
12:29:36,918 DEBUG akka.event.EventStream - Default Loggers started
12:29:36,930 INFO Remoting - Starting remoting
12:29:36,953 DEBUG org.jboss.netty.channel.socket.nio.SelectorUtil - Using select timeout of 500
12:29:36,953 DEBUG org.jboss.netty.channel.socket.nio.SelectorUtil - Epoll-bug workaround enabled = false
12:29:37,010 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@127.0.0.1:6123]
12:29:37,013 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManger web frontend
12:29:37,027 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2 for the web interface files
12:29:37,028 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Serving job manager log from /opt/flink-0.10.0/log/flink-kknauf-jobmanager-0-kknauf-ThinkPad-T440p.log
12:29:37,028 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Serving job manager stdout from /opt/flink-0.10.0/log/flink-kknauf-jobmanager-0-kknauf-ThinkPad-T440p.out
12:29:37,055 DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
12:29:37,057 DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:29:37,057 DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:29:37,057 DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:29:37,057 DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
12:29:37,058 DEBUG io.netty.util.internal.PlatformDependent - Java version: 8
12:29:37,058 DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:29:37,058 DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:29:37,058 DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:29:37,152 DEBUG io.netty.util.internal.PlatformDependent - Javassist: available
12:29:37,153 DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:29:37,153 DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:29:37,153 DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:29:37,154 DEBUG io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.handler.codec.http.router.RoutedMatcher
12:29:37,162 DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
12:29:37,172 DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
12:29:37,172 DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
12:29:37,181 DEBUG io.netty.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x13528f7b4ad258ce (took 0 ms)
12:29:37,202 DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
12:29:37,202 DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.threadLocalDirectBufferSize: 65536
12:29:37,204 DEBUG io.netty.util.NetUtil - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
12:29:37,204 DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128
12:29:37,208 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:8081
12:29:37,209 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor
12:29:37,211 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-3c5cb4ce-1fe3-4181-93f0-0bf598355482
12:29:37,212 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:57131 - max concurrent requests: 50 - max backlog: 1000
12:29:37,217 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager process reaper
12:29:37,218 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting embedded TaskManager for JobManager's LOCAL execution mode
12:29:37,220 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink@127.0.0.1:6123/user/jobmanager.
12:29:37,220 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive
12:29:37,223 INFO org.apache.flink.runtime.jobmanager.JobManager - JobManager akka.tcp://flink@127.0.0.1:6123/user/jobmanager was granted leadership with leader session ID None.
12:29:37,223 INFO org.apache.flink.runtime.taskmanager.TaskManager - Messages between TaskManager and JobManager have a max timeout of 100000 milliseconds
12:29:37,225 INFO org.apache.flink.runtime.taskmanager.TaskManager - Temporary file directory '/tmp': total 900 GB, usable 429 GB (47.67% usable)
12:29:37,236 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
12:29:37,315 INFO org.apache.flink.runtime.taskmanager.TaskManager - Using 0.7 of the currently free heap space for Flink managed heap memory (464 MB).
12:29:37,672 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-781d8416-1517-4f0e-983d-59aa9a39b75c for spill files.
12:29:37,674 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Starting TaskManager process reaper
12:29:37,674 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@127.0.0.1:6123/user/jobmanager on port 8081
12:29:37,674 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@127.0.0.1:6123/user/jobmanager:null.
12:29:37,676 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-3811fb9d-c567-44ca-8126-48565f9656bb
12:29:37,747 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager actor at akka://flink/user/taskmanager#810056198.
12:29:37,747 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager data connection information: localhost (dataPort=38939)
12:29:37,748 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager has 1 task slot(s).
12:29:37,748 INFO org.apache.flink.runtime.taskmanager.TaskManager - Memory usage stats: [HEAP: 551/736/736 MB, NON HEAP: 33/33/-1 MB (used/committed/max)]
12:29:37,750 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink@127.0.0.1:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
12:29:37,752 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at localhost (akka://flink/user/taskmanager) as bc212812d963df9dbc71f30a75aa6438. Current number of registered hosts is 1. Current number of alive task slots is 1.
12:29:37,753 INFO org.apache.flink.runtime.taskmanager.TaskManager - Successful registration at JobManager (akka://flink/user/jobmanager), starting network stack and library cache.
12:29:37,753 DEBUG org.apache.flink.runtime.io.network.NetworkEnvironment - Starting result partition manager and network connection manager
12:29:37,760 DEBUG org.apache.flink.runtime.io.network.NetworkEnvironment - Starting network connection manager
12:29:37,760 INFO org.apache.flink.runtime.taskmanager.TaskManager - Determined BLOB server address to be localhost/127.0.0.1:57131. Starting BLOB cache.
12:29:37,761 INFO org.apache.flink.runtime.blob.BlobCache - Created BLOB cache storage directory /tmp/blobStore-3cc248bc-376e-410d-997e-17be2481c2e9
12:29:38,263 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - TaskManager was triggered to register at JobManager, but is already registered
12:29:42,775 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
12:29:42,802 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Received hearbeat message from bc212812d963df9dbc71f30a75aa6438.
12:29:42,803 DEBUG org.apache.flink.runtime.instance.InstanceManager - Received heartbeat from TaskManager bc212812d963df9dbc71f30a75aa6438 @ localhost - 1 slots - URL: akka://flink/user/taskmanager
12:29:47,773 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
12:29:47,775 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Received hearbeat message from bc212812d963df9dbc71f30a75aa6438.
12:29:47,775 DEBUG org.apache.flink.runtime.instance.InstanceManager - Received heartbeat from TaskManager bc212812d963df9dbc71f30a75aa6438 @ localhost - 1 slots - URL: akka://flink/user/taskmanager
12:29:52,773 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
12:29:52,774 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Received hearbeat message from bc212812d963df9dbc71f30a75aa6438.
12:29:52,775 DEBUG org.apache.flink.runtime.instance.InstanceManager - Received heartbeat from TaskManager bc212812d963df9dbc71f30a75aa6438 @ localhost - 1 slots - URL: akka://flink/user/taskmanager
12:29:53,477 DEBUG io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.handler.codec.http.HttpRequestMatcher
12:29:53,487 DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:29:53,487 DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:29:53,494 DEBUG io.netty.util.internal.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.handler.codec.http.HttpObjectMatcher
12:29:53,497 DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
12:29:53,499 DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:29:53,509 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /index.html
12:29:53,515 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/index.html'
12:29:53,536 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /css/vendor.css
12:29:53,538 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /css/index.css
12:29:53,539 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/css/index.css'
12:29:53,541 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/css/vendor.css'
12:29:53,544 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /js/vendor.js
12:29:53,544 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /js/index.js
12:29:53,546 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/js/index.js'
12:29:53,581 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/js/vendor.js'
12:29:53,606 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /images/flink-logo.png
12:29:53,606 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/images/flink-logo.png'
12:29:53,806 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /partials/jobmanager/index.html
12:29:53,807 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/partials/jobmanager/index.html'
12:29:53,827 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Loading missing file from classloader: /partials/jobmanager/config.html
12:29:53,827 DEBUG org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler - Responding with file '/tmp/flink-web-71af5001-d76c-4180-9c46-bcb6717052c2/partials/jobmanager/config.html'
12:29:57,773 DEBUG org.apache.flink.runtime.taskmanager.TaskManager - Sending heartbeat to JobManager
12:29:57,775 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Received hearbeat message from bc212812d963df9dbc71f30a75aa6438.
12:29:57,775 DEBUG org.apache.flink.runtime.instance.InstanceManager - Received heartbeat from TaskManager bc212812d963df9dbc71f30a75aa6438 @ localhost - 1 slots - URL: akka://flink/user/taskmanager
12:30:00,895 DEBUG akka.serialization.Serialization(akka://flink) - Using serializer[akka.serialization.JavaSerializer] for message [akka.actor.ActorIdentity]
12:30:00,922 DEBUG akka.serialization.Serialization(akka://flink) - Using serializer[akka.serialization.JavaSerializer] for message [java.lang.Integer]
12:30:00,930 DEBUG org.apache.flink.runtime.blob.BlobServerConnection - Received PUT request for content addressable BLOB
12:30:00,982 INFO org.apache.flink.runtime.jobmanager.JobManager - Submitting job 78df7141119c342542aafa00964434f9 (Flink Vs Spark - Flink Static).
12:30:00,998 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Running initialization on master for job 78df7141119c342542aafa00964434f9 (Flink Vs Spark - Flink Static).
12:30:01,026 DEBUG org.apache.flink.runtime.jobmanager.JobManager - Adding 4 vertices from job graph 78df7141119c342542aafa00964434f9 (Flink Vs Spark - Flink Static).
12:30:01,026 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 4 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results.
12:30:01,034 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-default configuration file
12:30:01,034 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-site configuration file
12:30:01,035 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Trying to load HDFS class Hadoop 2.x style.
12:30:01,038 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Falling back to loading HDFS class old Hadoop style. Looking for HDFS class configuration entry 'fs.hdfs.impl'.
12:30:01,051 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Loaded HDFS class 'org.apache.hadoop.hdfs.DistributedFileSystem' as specified in configuration.
12:30:01,051 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-default configuration file
12:30:01,051 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-site configuration file
12:30:01,065 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Flink supports s3 with the Hadoop file system wrapper, impl class org.apache.hadoop.fs.s3.S3FileSystem
12:30:01,065 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-default configuration file
12:30:01,065 DEBUG org.apache.flink.runtime.fs.hdfs.HadoopFileSystem - Cannot find hdfs-site configuration file
12:30:01,086 ERROR org.apache.flink.runtime.jobmanager.JobManager - Failed to submit job 78df7141119c342542aafa00964434f9 (...)
org.apache.flink.runtime.JobException: Creating the input splits caused an error: The given file URI (s3://com.tngtech.knaufk.ireland/flink_input/detail_table.csv) points to the HDFS NameNode at com.tngtech.knaufk.ireland, but the File System could not be initialized with that address: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:168)
at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:640)
at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:878)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:341)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.IOException: The given file URI (s3://com.tngtech.knaufk.ireland/flink_input/detail_table.csv) points to the HDFS NameNode at com.tngtech.knaufk.ireland, but the File System could not be initialized with that address: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:337)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:236)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:309)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:449)
at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:152)
... 25 more
Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:66)
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.initialize(Jets3tFileSystemStore.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy6.initialize(Unknown Source)
at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:77)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:321)
... 30 more
