Yes, we’re onto the exactly-once ; trying to write RCFiles (Parquet and ORCFiles are not compatible because of their footer).
It seems to be working perfectly. As expected, Flink is falling back to .valid-length metadata on HDFS 2.6 (and 2.3). From: Robert Metzger [mailto:rmetz...@apache.org] Sent: mercredi 14 octobre 2015 17:23 To: user@flink.apache.org Subject: Re: Building Flink with hadoop 2.6 Great. We are shading curator now into a different location, that's why you can't find it anymore. I suspect you're trying out our new exactly-once filesystem sinks. Please let us know how well its working for you and if you're missing something. Its a pretty new feature :) Also note that you can use the fs sinks with hadoop versions below 2.7.0, then we'll write some metadata containing the valid offsets. On Wed, Oct 14, 2015 at 5:18 PM, Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>> wrote: Yes … You’re right. Anyway, adding the log4j jar solved the issue and our app is working properly, thanks ! About curator, I just observed that it was not there anymore when comparing the old and new fatjars. But it’s probably now in another dependency, anyway there is no curator-related error so it just probably moved. Thanks ! Gwen’ From: Robert Metzger [mailto:rmetz...@apache.org<mailto:rmetz...@apache.org>] Sent: mercredi 14 octobre 2015 17:06 To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Building Flink with hadoop 2.6 One more thing regarding the truncate method: Its supported as of Hadoop 2.7.0 (https://issues.apache.org/jira/browse/HDFS-3107) On Wed, Oct 14, 2015 at 5:00 PM, Robert Metzger <rmetz...@apache.org<mailto:rmetz...@apache.org>> wrote: Ah, I know what's causing this issue. In the latest 0.10-SNAPSHOT, we have removed log4j from the fat jar. Can you copy everything from the lib/ folder from your maven build into the lib/ folder of your flink installation? Log4j is now in a separate jar in the lib/ folder . What about the curator dependency issue? On Wed, Oct 14, 2015 at 4:56 PM, Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>> wrote: The first class that it can not find is : org.apache.log4j.Level The org.apache.log4j package is not present in the fat jar I get from the mvn command, but it is in the one you distributed on your website. From: Robert Metzger [mailto:rmetz...@apache.org<mailto:rmetz...@apache.org>] Sent: mercredi 14 octobre 2015 16:54 To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Building Flink with hadoop 2.6 Great. Which classes can it not find at runtime? I'll try to build and run Flink with exactly the command you've provided. On Wed, Oct 14, 2015 at 4:49 PM, Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>> wrote: Hi Robert ! I’m using ” mvn clean install -DskipTests -Dhadoop.version=2.6.0 “. From: Robert Metzger [mailto:rmetz...@apache.org<mailto:rmetz...@apache.org>] Sent: mercredi 14 octobre 2015 16:47 To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Building Flink with hadoop 2.6 Hi Gwen, can you tell us the "mvn" command you're using for building Flink? On Wed, Oct 14, 2015 at 4:37 PM, Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com<mailto:gwenhael.pasqui...@ericsson.com>> wrote: Hi ; We need to test some things with flink and hadoop 2.6 (the trunc method). We’ve set up a build task on our Jenkins and everything seem okay. However when we replace the original jar from your 0.10-SNAPSHOT distribution by ours there are some missing dependencies (log4j, curator, and maybe others) and we get some ClassNotFoundException at runtime. Are we missing some build parameters ? Thanks in advance, B.R.
