[
https://issues.apache.org/jira/browse/HIVE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773932#comment-13773932
]
Xuefu Zhang commented on HIVE-5318:
-----------------------------------
[~ashutoshc] Test was planed when patch is submitted.
As to the nullness of these lists, your assumption might be well correct, but
that's not what I see in the code.
Here is the caller code in ImportSemanticAnalyzer.
{code}
tblDesc = new CreateTableDesc(
table.getTableName(),
false, // isExternal: set to false here, can be overwritten by the
// IMPORT stmt
table.getSd().getCols(),
table.getPartitionKeys(),
table.getSd().getBucketCols(),
table.getSd().getSortCols(),
table.getSd().getNumBuckets(),
null, null, null, null, null, // these 5 delims passed as serde
params
null, // comment passed as table params
table.getSd().getInputFormat(),
table.getSd().getOutputFormat(),
null, // location: set to null here, can be
// overwritten by the IMPORT stmt
table.getSd().getSerdeInfo().getSerializationLib(),
null, // storagehandler passed as table params
table.getSd().getSerdeInfo().getParameters(),
table.getParameters(), false,
(null == table.getSd().getSkewedInfo()) ? null :
table.getSd().getSkewedInfo()
.getSkewedColNames(),
(null == table.getSd().getSkewedInfo()) ? null :
table.getSd().getSkewedInfo()
.getSkewedColValues());
{code}
>From the snippet we can see that, it's possible that the last two lists can be
>null. Also, the partition columns was passed from the thrift "table" object,
>for which null is clearly a valid value.
For reference, this is the metadata for an exported, simple table with two
columns and two rows of data:
{code}
{"partitions":[],"table":"{\"1\":{\"str\":\"j1_41\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"johndee\"},\"4\":{\"i32\":1371900915},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"a\"},\"2\":{\"str\":\"string\"}},{\"1\":{\"str\":\"b\"},\"2\":{\"str\":\"int\"}}]},\"2\":{\"str\":\"hdfs://hivebase01:8020/user/hive/warehouse/j1_41\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.TextInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":-1},\"7\":{\"rec\":{\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",2,{\"serialization.format\":\",\",\"field.delim\":\",\"}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",1,{\"transient_lastDdlTime\":\"1371900931\"}]},\"12\":{\"str\":\"MANAGED_TABLE\"}}","version":"0.1"}
{code}
This piece of meta data contains no partition columns, or skewedkey/values, etc.
Could you clarify if you meant to say that the list should not null but with
zero element? For unknown reason, the code doesn't reflect that either. for
instance, Utilities.getFieldSchemaString() has code to handle a null list of
partition columns.
Any further insight is appreciated.
> Import Throws Error when Importing from a table export Hive 0.9 to Hive 0.10
> ----------------------------------------------------------------------------
>
> Key: HIVE-5318
> URL: https://issues.apache.org/jira/browse/HIVE-5318
> Project: Hive
> Issue Type: Bug
> Components: Import/Export
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Brad Ruderman
> Assignee: Xuefu Zhang
> Priority: Critical
> Attachments: HIVE-5318.patch
>
>
> When Exporting hive tables using the hive command in Hive 0.9 "EXPORT table
> TO 'hdfs_path'" then importing to another hive 0.10 instance using "IMPORT
> FROM 'hdfs_path'", hive throws this error:
> 13/09/18 13:14:02 ERROR ql.Driver: FAILED: SemanticException Exception while
> processing
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception while processing
> at
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.analyzeInternal(ImportSemanticAnalyzer.java:277)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:459)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:938)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:706)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.<init>(ArrayList.java:131)
> at
> org.apache.hadoop.hive.ql.plan.CreateTableDesc.<init>(CreateTableDesc.java:128)
> at
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.analyzeInternal(ImportSemanticAnalyzer.java:99)
> ... 16 more
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=compile
> start=1379535241411 end=1379535242332 duration=921>
> 13/09/18 13:14:02 INFO ql.Driver: <PERFLOG method=releaseLocks>
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=releaseLocks
> start=1379535242332 end=1379535242332 duration=0>
> 13/09/18 13:14:02 INFO ql.Driver: <PERFLOG method=releaseLocks>
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=releaseLocks
> start=1379535242333 end=1379535242333 duration=0>
> This is probably a critical blocker for people who are trying to test Hive
> 0.10 in their staging environments prior to the upgrade from 0.9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
