Hello. I'm running Hadoop 2.2.0 and Hive 0.12.0.
Hadoop supports encryption of client writes to the data node, as described under 'Data Transfer Protocol' here: http://hortonworks.com/blog/wire-encryption-hadoop/. The encryption appears to work for simple hadoop commands (copyFromLocal, load data infile, etc.) However, when I attempt to execute a Hive job I receive the following errors: java.lang.RuntimeException: Cannot serialize object at org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652) at java.beans.XMLEncoder.writeln(XMLEncoder.java:489) at java.beans.XMLEncoder.outputValue(XMLEncoder.java:543) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687) at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687) at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687) at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687) at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552) at java.beans.XMLEncoder.flush(XMLEncoder.java:398) at java.beans.XMLEncoder.close(XMLEncoder.java:429) at org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:667) at org.apache.hadoop.hive.ql.exec.Utilities.serializePlan(Utilities.java:610) at org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:495) at org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:479) at org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:472) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:383) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:731) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.nio.channels.ClosedChannelException at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1317) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) at java.io.FilterOutputStream.write(FilterOutputStream.java:80) at java.beans.XMLEncoder.writeln(XMLEncoder.java:485) ... 38 more This seems to imply that Hive does not have encryption enabled and thus objects cannot be serialized to the encrypted communication channel. Does Hive support DN transfer protocol encryption? If so, is there documentation somewhere on usage? I did find several pages on Hive encryption, but these merely refer to Hive JDBC connection encryption. Regards, Bryan Jeffrey
