That log does not appear. It looks like we have egg and chicken issue.
2019-02-15 16:49:15,045 DEBUG org.apache.hadoop.hdfs.DFSClient
- Connecting to datanode 10.246.221.10:50010
2019-02-15 16:49:15,045 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient -
SASL client skipping handshake in unsecured configuration for
addr = /10.246.221.10, datanodeId = DatanodeInfoWithStorage[
10.246.221.10:50010,DS-c57a7667-f697-4f03-9fb1-532c5b82a9e8,DISK]
2019-02-15 16:49:15,072 DEBUG
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory -
Instantiating for file system scheme hdfs Hadoop File System
org.apache.hadoop.hdfs.DistributedFileSystem
2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
- dfs.client.use.legacy.blockreader.local = false
2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
- dfs.client.read.shortcircuit = false
2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
- dfs.client.domain.socket.data.traffic = false
2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
- dfs.domain.socket.path =
2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.io.retry.RetryUtils
- multipleLinearRandomRetry = null
2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.ipc.Client
- getting client out of cache:
org.apache.hadoop.ipc.Client@31920ade
2019-02-15 16:49:15,076 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil -
DataTransferProtocol not using SaslPropertiesResolver, no QOP found in
configuration for dfs.data.transfer.protection
2019-02-15 16:49:15,080 INFO
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets - Subtask
3 initializing its state (max part counter=58).
2019-02-15 16:49:15,081 DEBUG
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets - Subtask
3 restoring: BucketState for
bucketId=ls_kraken_events/dt=2019-02-14/evt=ad_fill and
bucketPath=hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill,
has open part file created @ 1550247946437
2019-02-15 16:49:15,085 DEBUG org.apache.hadoop.ipc.Client
- IPC Client (1270836494) connection to
nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root sending #56
2019-02-15 16:49:15,188 DEBUG org.apache.hadoop.ipc.Client
- IPC Client (1270836494) connection to
nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root got value #56
2019-02-15 16:49:15,196 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Custom Source -> (Sink: Unnamed, Process ->
Timestamps/Watermarks) (4/4) (f73403ac4763c99e6a244cba3797f7e9) switched
from RUNNING to FAILED.
java.io.IOException: Missing data in tmp file:
hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill/
.part-3-32.inprogress.da2a75d1-0c83-47bc-9c83-950360c55c86
at
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.<init>(HadoopRecoverableFsDataOutputStream.java:93)
I do see
2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Current Hadoop/Kerberos user: root
2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13
2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Maximum heap size: 1204 MiBytes
2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- JAVA_HOME: /docker-java-home
2019-02-15 16:47:33,585 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Hadoop version: 2.7.5
which has to be expected given that we are running the hadoop27flink 1.7.1
version.
Does it make sense to go with a hadoop less version and inject the required
jar files ? Has that been done by anyone ?
On Fri, Feb 15, 2019 at 2:33 AM Yun Tang <myas...@live.com> wrote:
> Hi
>
> When 'RollingSink' try to initialize state, it would first check current
> file system supported truncate method. If file system not supported, it
> would use another work-around solution, which means you should not meet the
> problem. Otherwise 'RollingSink' thought and found the reflection method of
> 'truncate' while the file system actually not support. You could try to
> open DEBUG level to see whether log below could be printed:
> Truncate not found. Will write a file with suffix '.valid-length' and
> prefix '_' to specify how many bytes in a bucket are valid.
>
> However, from your second email, the more serious problem should be using
> 'Buckets' with Hadoop-2.6. From what I know the `RecoverableWriter` within
> 'Buckets' can only support Hadoop-2.7+ , I'm not sure whether existed work
> around solution.
>
> Best
> Yun Tang
> ------------------------------
> *From:* Vishal Santoshi <vishal.santo...@gmail.com>
> *Sent:* Friday, February 15, 2019 3:43
> *To:* user
> *Subject:* Re: StandAlone job on k8s fails with "Unknown method truncate"
> on restore
>
> And yes cannot work with RollingFleSink for hadoop 2.6 release of 1.7.1
> b'coz of this.
>
> java.lang.UnsupportedOperationException: Recoverable writers on Hadoop are
> only supported for HDFS and for Hadoop version 2.7 or newer
> at
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:57)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
>
>
> Any work around ?
>
>
> On Thu, Feb 14, 2019 at 1:42 PM Vishal Santoshi <vishal.santo...@gmail.com>
> wrote:
>
> The job uses a RolllingFileSink to push data to hdfs. Run an HA standalone
> cluster on k8s,
>
> * get the job running
> * kill the pod.
>
> The k8s deployment relaunches the pod but fails with
>
> java.io.IOException: Missing data in tmp file:
> hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill/.part-2-16.inprogress.449e8668-e886-4f89-b5f6-45ac68e25987
>
>
> Unknown method truncate called on
> org.apache.hadoop.hdfs.protocol.ClientProtocol protocol.
>
>
> The file does exist. We work with hadoop 2.6 , which does no have
> truncate. The previous version would see that "truncate" was not supported
> and drop a length file for the ,inprogress file and rename it to a valid
> part file.
>
>
>
> Is this a known issue ?
>
>
> Regards.
>
>
>
>
>
