Thanks for the information. Without setting such parameter explicitly, is there any possibility that it may work intermittently?
From: Dian Fu <dian0511...@gmail.com> Sent: Tuesday, October 29, 2019 7:12 AM To: V N, Suchithra (Nokia - IN/Bangalore) <suchithra....@nokia.com> Cc: user@flink.apache.org Subject: Re: Flink 1.8.1 HDFS 2.6.5 issue I guess this is a bug in Hadoop 2.6.5 and has been fixed in Hadoop 2.8.0 [1]. You can work around it by explicitly setting the configration "hadoop.security.crypto.codec.classes.aes.ctr.nopadding" as "org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec". [1] https://issues.apache.org/jira/browse/HADOOP-11711 在 2019年10月28日，下午8:59，V N, Suchithra (Nokia - IN/Bangalore) <suchithra....@nokia.com<mailto:suchithra....@nokia.com>> 写道： Hi, >From debug logs I could see below logs in taskmanager. Please have a look. org.apache.hadoop.ipc.ProtobufRpcEngine Call: addBlock took 372ms"} org.apache.hadoop.ipc.ProtobufRpcEngine Call: addBlock took 372ms"} org.apache.hadoop.hdfs.DFSClient pipeline = 10.76.113.216:1044"} org.apache.hadoop.hdfs.DFSClient pipeline = 10.76.113.216:1044"} org.apache.hadoop.hdfs.DFSClient Connecting to datanode 10.76.113.216:1044"} org.apache.hadoop.hdfs.DFSClient Connecting to datanode 10.76.113.216:1044"} org.apache.hadoop.hdfs.DFSClient Send buf size 131072"} org.apache.hadoop.hdfs.DFSClient Send buf size 131072"} o.a.h.h.p.d.sasl.SaslDataTransferClient SASL client doing encrypted handshake for addr = /10.76.113.216, datanodeId = 10.76.113.216:1044"} o.a.h.h.p.d.sasl.SaslDataTransferClient SASL client doing encrypted handshake for addr = /10.76.113.216, datanodeId = 10.76.113.216:1044"} o.a.h.h.p.d.sasl.SaslDataTransferClient Client using encryption algorithm 3des"} o.a.h.h.p.d.sasl.SaslDataTransferClient Client using encryption algorithm 3des"} o.a.h.h.p.d.sasl.DataTransferSaslUtil Verifying QOP, requested QOP = [auth-conf], negotiated QOP = auth-conf"} o.a.h.h.p.d.sasl.DataTransferSaslUtil Verifying QOP, requested QOP = [auth-conf], negotiated QOP = auth-conf"} o.a.h.h.p.d.sasl.DataTransferSaslUtil Creating IOStreamPair of CryptoInputStream and CryptoOutputStream."} o.a.h.h.p.d.sasl.DataTransferSaslUtil Creating IOStreamPair of CryptoInputStream and CryptoOutputStream."} o.apache.hadoop.util.PerformanceAdvisory No crypto codec classes with cipher suite configured."} o.apache.hadoop.util.PerformanceAdvisory No crypto codec classes with cipher suite configured."} org.apache.hadoop.hdfs.DFSClient DataStreamer Exception"} java.lang.NullPointerException: null at org.apache.hadoop.crypto.CryptoInputStream.<init>(CryptoInputStream.java:132) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:489) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:298) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:210) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:182) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1409) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587) Regards, Suchithra From: Dian Fu <dian0511...@gmail.com<mailto:dian0511...@gmail.com>> Sent: Monday, October 28, 2019 5:40 PM To: V N, Suchithra (Nokia - IN/Bangalore) <suchithra....@nokia.com<mailto:suchithra....@nokia.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink 1.8.1 HDFS 2.6.5 issue It seems that the CryptoCodec is null from the exception stack trace. This may occur when "hadoop.security.crypto.codec.classes.aes.ctr.nopadding" is misconfigured. You could change the log level to "DEBUG" and it will show more detailed information about why CryptoCodec is null. 在 2019年10月28日，下午7:14，V N, Suchithra (Nokia - IN/Bangalore) <suchithra....@nokia.com<mailto:suchithra....@nokia.com>> 写道： Hi, I am trying to execute Wordcount.jar in Flink 1.8.1 with Hadoop version 2.6.5. HDFS is enabled with Kerberos+SSL. While writing output to HDFS, facing the below exception and job will be failed. Please let me know if any suggestions to debug this issue. Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) ... 21 more Caused by: java.io.IOException: DataStreamer Exception: at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:695) Caused by: java.lang.NullPointerException at org.apache.hadoop.crypto.CryptoInputStream.<init>(CryptoInputStream.java:132) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.createStreamPair(DataTransferSaslUtil.java:345) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:489) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:298) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:210) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:182) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1409) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1357) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:587) Regards, Suchithra
