And yes cannot work with RollingFleSink for hadoop 2.6 release of 1.7.1
b'coz of this.
java.lang.UnsupportedOperationException: Recoverable writers on Hadoop
are only supported for HDFS and for Hadoop version 2.7 or newer
at
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:57)
at
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
Any work around ?
On Thu, Feb 14, 2019 at 1:42 PM Vishal Santoshi <vishal.santo...@gmail.com>
wrote:
> The job uses a RolllingFileSink to push data to hdfs. Run an HA standalone
> cluster on k8s,
>
> * get the job running
> * kill the pod.
>
> The k8s deployment relaunches the pod but fails with
>
> java.io.IOException: Missing data in tmp file:
> hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill/.part-2-16.inprogress.449e8668-e886-4f89-b5f6-45ac68e25987
>
>
> Unknown method truncate called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol.
>
>
> The file does exist. We work with hadoop 2.6 , which does no have
> truncate. The previous version would see that "truncate" was not supported
> and drop a length file for the ,inprogress file and rename it to a valid
> part file.
>
>
>
> Is this a known issue ?
>
>
> Regards.
>
>
>
>
>
