Ah, I see. Maybe it would make sense then for you to use the latest Hadoop
version we are supporting. This way, you get the most recent Hadoop S3 file
system implementation.
Note that there might be an issue with starting Flink 0.10.0 for Hadoop
2.7.0. We'll fix it with Flink 0.10.1.
But if everything is working fine ... it might make sense not to change it
now ("never change a running system").
On Sat, Nov 21, 2015 at 3:24 PM, Konstantin Knauf <
konstantin.kn...@tngtech.com> wrote:
> Hi Robert,
>
> I am basically only reading from Kafka and S3 and writing to S3 in this
> job. So I am using the Hadoop S3 FileSystem classes, but that's it.
>
> Cheers,
>
> Konstantin
>
>
> On 21.11.2015 15:16, Robert Metzger wrote:
> > Hi,
> >
> > great to hear that its working. I've updated the documentation (for 1.0)
> > and made the word directory bold ;)
> >
> > You should try to match your Hadoop version as closely as possible.
> > Are you not using HDFS at all? Then it doesn't matter which version of
> > Flink you are downloading.
> > When using Hadoop 2.x then I'd recommend at least a Flink version for
> > Hadoop 2.3.0
> >
> >
> > On Sat, Nov 21, 2015 at 3:13 PM, Konstantin Knauf
> > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>>
> wrote:
> >
> > Hi Robert,
> >
> > thanks a lot, it's working now. Actually, it also says "directory" in
> > the description. So I should have known :/
> >
> > On additional question though. If I use the flink binary for Hadoop
> > 1.2.1 and run flink in standalone mode, should I use the *-hadoop1
> > dependencies even If I am not interacting with HDFS 1.x?
> >
> > Cheers,
> >
> > Konstantin
> >
> > On 21.11.2015 14:52, Robert Metzger wrote:
> > > Hi,
> > >
> > > It seems that you've set the "fs.hdfs.hadoopconf" configuration
> > > parameter to a file. I think you have to set it the directory
> containing
> > > the configuration.
> > > Sorry, I know that's not very intuitive, but in Hadoop the
> settings for
> > > in different files (hdfs|yarn|core)-site.xml.
> > >
> > >
> > > On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf
> > > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com
> >
> > <mailto:konstantin.kn...@tngtech.com
> > <mailto:konstantin.kn...@tngtech.com>>> wrote:
> > >
> > > Hi Ufuk,
> > >
> > > sorry for not getting back to you for so long, and thanks for
> your
> > > answer. The problem persists unfortunately. Running the job
> from the IDE
> > > works (with core-site.xml on classpath), running it in local
> standalone
> > > mode does not. AccessKeyID and SecretAccesKey are not found.
> > >
> > > Attached the jobmanager log on DEBUG level. The core-site.xml
> is
> > > definitely at the configured location.
> > >
> > > I am now on version 0.10.0 and using the binaries for Hadoop
> 1.2.1 to
> > > run the jar in local mode. Do I have to use the Hadoop 2.x
> version for
> > > this to work? I have put hadoop-common-2.3.jar into the flink
> lib
> > > folder.
> > >
> > > I don't know if it is relevant (but it seems to be related),
> when I run
> > > the job from my IDE I get the warning:
> > >
> > > 2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load
> > > native-hadoop library for your platform... using builtin-java
> classes
> > > where applicable
> > >
> > > Cheers and thank you,
> > >
> > > Konstantin
> > >
> > >
> > > On 14.10.2015 11:44, Ufuk Celebi wrote:
> > > >
> > > >> On 10 Oct 2015, at 22:59, snntr <
> konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>
> > > <mailto:konstantin.kn...@tngtech.com <mailto:
> konstantin.kn...@tngtech.com>>>
> > wrote:
> > > >>
> > > >> Hey everyone,
> > > >>
> > > >> I was having the same problem with S3 and found this thread
> very
> > > useful.
> > > >> Everything works fine now, when I start Flink from my IDE,
> but
> > > when I run
> > > >> the jar in local mode I keep getting
> > > >>
> > > >> java.lang.IllegalArgumentException: AWS Access Key ID and
> Secret
> > > Access Key
> > > >> must be specified as the username or password
> (respectively) of a
> > > s3n URL,
> > > >> or by setting the fs.s3n.awsAccessKeyId or
> fs.s3n.awsSecretAccessKey
> > > >> properties (respectively).
> > > >>
> > > >> I have set fs.hdfs.hadoopconf to point to a core-site.xml
> on my local
> > > >> machine with the required properties. What am I missing?
> > > >>
> > > >> Any advice is highly appreciated ;)
> > > >
> > > > This looks like a problem with picking up the Hadoop config.
> Can
> > > you look into the logs to check whether the configuration is
> picked
> > > up? Change the log settings to DEBUG in log/log4j.properties
> for
> > > this. And can you provide the complete stack trace?
> > > >
> > > > – Ufuk
> > > >
> > > >
> > >
> > > --
> > > Konstantin Knauf * konstantin.kn...@tngtech.com <mailto:
> konstantin.kn...@tngtech.com>
> > > <mailto:konstantin.kn...@tngtech.com
> > <mailto:konstantin.kn...@tngtech.com>> * +49-174-3413182
> > <tel:%2B49-174-3413182>
> > > <tel:%2B49-174-3413182>
> > > TNG Technology Consulting GmbH, Betastr. 13a, 85774
> Unterföhring
> > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
> > Dahlke
> > > Sitz: Unterföhring * Amtsgericht München * HRB 135082
> > >
> > >
> >
> > --
> > Konstantin Knauf * konstantin.kn...@tngtech.com
> > <mailto:konstantin.kn...@tngtech.com> * +49-174-3413182
> > <tel:%2B49-174-3413182>
> > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> > Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >
> >
>
> --
> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
