Hi Robert, I am basically only reading from Kafka and S3 and writing to S3 in this job. So I am using the Hadoop S3 FileSystem classes, but that's it.
Cheers, Konstantin On 21.11.2015 15:16, Robert Metzger wrote: > Hi, > > great to hear that its working. I've updated the documentation (for 1.0) > and made the word directory bold ;) > > You should try to match your Hadoop version as closely as possible. > Are you not using HDFS at all? Then it doesn't matter which version of > Flink you are downloading. > When using Hadoop 2.x then I'd recommend at least a Flink version for > Hadoop 2.3.0 > > > On Sat, Nov 21, 2015 at 3:13 PM, Konstantin Knauf > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com>> wrote: > > Hi Robert, > > thanks a lot, it's working now. Actually, it also says "directory" in > the description. So I should have known :/ > > On additional question though. If I use the flink binary for Hadoop > 1.2.1 and run flink in standalone mode, should I use the *-hadoop1 > dependencies even If I am not interacting with HDFS 1.x? > > Cheers, > > Konstantin > > On 21.11.2015 14:52, Robert Metzger wrote: > > Hi, > > > > It seems that you've set the "fs.hdfs.hadoopconf" configuration > > parameter to a file. I think you have to set it the directory containing > > the configuration. > > Sorry, I know that's not very intuitive, but in Hadoop the settings for > > in different files (hdfs|yarn|core)-site.xml. > > > > > > On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf > > <konstantin.kn...@tngtech.com <mailto:konstantin.kn...@tngtech.com> > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>> wrote: > > > > Hi Ufuk, > > > > sorry for not getting back to you for so long, and thanks for your > > answer. The problem persists unfortunately. Running the job from > the IDE > > works (with core-site.xml on classpath), running it in local > standalone > > mode does not. AccessKeyID and SecretAccesKey are not found. > > > > Attached the jobmanager log on DEBUG level. The core-site.xml is > > definitely at the configured location. > > > > I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 > to > > run the jar in local mode. Do I have to use the Hadoop 2.x version > for > > this to work? I have put hadoop-common-2.3.jar into the flink lib > > folder. > > > > I don't know if it is relevant (but it seems to be related), when I > run > > the job from my IDE I get the warning: > > > > 2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load > > native-hadoop library for your platform... using builtin-java > classes > > where applicable > > > > Cheers and thank you, > > > > Konstantin > > > > > > On 14.10.2015 11:44, Ufuk Celebi wrote: > > > > > >> On 10 Oct 2015, at 22:59, snntr <konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>>> > wrote: > > >> > > >> Hey everyone, > > >> > > >> I was having the same problem with S3 and found this thread very > > useful. > > >> Everything works fine now, when I start Flink from my IDE, but > > when I run > > >> the jar in local mode I keep getting > > >> > > >> java.lang.IllegalArgumentException: AWS Access Key ID and Secret > > Access Key > > >> must be specified as the username or password (respectively) of a > > s3n URL, > > >> or by setting the fs.s3n.awsAccessKeyId or > fs.s3n.awsSecretAccessKey > > >> properties (respectively). > > >> > > >> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my > local > > >> machine with the required properties. What am I missing? > > >> > > >> Any advice is highly appreciated ;) > > > > > > This looks like a problem with picking up the Hadoop config. Can > > you look into the logs to check whether the configuration is picked > > up? Change the log settings to DEBUG in log/log4j.properties for > > this. And can you provide the complete stack trace? > > > > > > – Ufuk > > > > > > > > > > -- > > Konstantin Knauf * konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> > > <mailto:konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com>> * +49-174-3413182 > <tel:%2B49-174-3413182> > > <tel:%2B49-174-3413182> > > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert > Dahlke > > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > > > > > -- > Konstantin Knauf * konstantin.kn...@tngtech.com > <mailto:konstantin.kn...@tngtech.com> * +49-174-3413182 > <tel:%2B49-174-3413182> > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > > -- Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082
