Hi, great to hear that its working. I've updated the documentation (for 1.0) and made the word directory bold ;)
You should try to match your Hadoop version as closely as possible. Are you not using HDFS at all? Then it doesn't matter which version of Flink you are downloading. When using Hadoop 2.x then I'd recommend at least a Flink version for Hadoop 2.3.0 On Sat, Nov 21, 2015 at 3:13 PM, Konstantin Knauf < konstantin.kn...@tngtech.com> wrote: > Hi Robert, > > thanks a lot, it's working now. Actually, it also says "directory" in > the description. So I should have known :/ > > On additional question though. If I use the flink binary for Hadoop > 1.2.1 and run flink in standalone mode, should I use the *-hadoop1 > dependencies even If I am not interacting with HDFS 1.x? > > Cheers, > > Konstantin > > On 21.11.2015 14:52, Robert Metzger wrote: > > Hi, > > > > It seems that you've set the "fs.hdfs.hadoopconf" configuration > > parameter to a file. I think you have to set it the directory containing > > the configuration. > > Sorry, I know that's not very intuitive, but in Hadoop the settings for > > in different files (hdfs|yarn|core)-site.xml. > > > > > > On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf > > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>> > wrote: > > > > Hi Ufuk, > > > > sorry for not getting back to you for so long, and thanks for your > > answer. The problem persists unfortunately. Running the job from the > IDE > > works (with core-site.xml on classpath), running it in local > standalone > > mode does not. AccessKeyID and SecretAccesKey are not found. > > > > Attached the jobmanager log on DEBUG level. The core-site.xml is > > definitely at the configured location. > > > > I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to > > run the jar in local mode. Do I have to use the Hadoop 2.x version > for > > this to work? I have put hadoop-common-2.3.jar into the flink lib > > folder. > > > > I don't know if it is relevant (but it seems to be related), when I > run > > the job from my IDE I get the warning: > > > > 2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load > > native-hadoop library for your platform... using builtin-java classes > > where applicable > > > > Cheers and thank you, > > > > Konstantin > > > > > > On 14.10.2015 11:44, Ufuk Celebi wrote: > > > > > >> On 10 Oct 2015, at 22:59, snntr <konstantin.kn...@tngtech.com > > <mailto:konstantin.kn...@tngtech.com>> wrote: > > >> > > >> Hey everyone, > > >> > > >> I was having the same problem with S3 and found this thread very > > useful. > > >> Everything works fine now, when I start Flink from my IDE, but > > when I run > > >> the jar in local mode I keep getting > > >> > > >> java.lang.IllegalArgumentException: AWS Access Key ID and Secret > > Access Key > > >> must be specified as the username or password (respectively) of a > > s3n URL, > > >> or by setting the fs.s3n.awsAccessKeyId or > fs.s3n.awsSecretAccessKey > > >> properties (respectively). > > >> > > >> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my > local > > >> machine with the required properties. What am I missing? > > >> > > >> Any advice is highly appreciated ;) > > > > > > This looks like a problem with picking up the Hadoop config. Can > > you look into the logs to check whether the configuration is picked > > up? Change the log settings to DEBUG in log/log4j.properties for > > this. And can you provide the complete stack trace? > > > > > > – Ufuk > > > > > > > > > > -- > > Konstantin Knauf * konstantin.kn...@tngtech.com > > <mailto:konstantin.kn...@tngtech.com> * +49-174-3413182 > > <tel:%2B49-174-3413182> > > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > > > > > -- > Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 >
