[
https://issues.apache.org/jira/browse/HIVE-10722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-10722:
------------------------------------
Description:
There can be directories in HDFS containing unprintable characters; when doing
hadoop fs -ls, these characters are not even visible, and can only be seen for
example if output is piped thru od.
When these are loaded, they are stored in e.g. mysql as "?" (literal question
mark, findable via LIKE '%?%' in db) and show accordingly in Hive.
However, datanucleus appears to encode it as %3F; this causes the partition to
be unusable - it cannot be dropped, and other operations like drop table get
stuck (didn't investigate in detail why; drop table got unstuck as soon as
partition is removed from metastore).
We should probably have a 2-way option for such cases - error out on load
(default), or convert to '?' (that should actually work)/drop such characters.
We should also check if partitions with '?' inserted explicitly work at all
with datanucleus.
was:
There can be directories in HDFS containing unprintable characters; when doing
hadoop fs -ls, these are not even visible, and can only be seen for example if
output is piped thru od.
When these are loaded, they are stored in e.g. mysql as "?" (literal question
mark, findable via LIKE '%?%' in db) and show accordingly in Hive.
However, datanucleus appears to encode it as %3F; this causes the partition to
be unusable - it cannot be dropped, and other operations like drop table get
stuck (didn't investigate in detail why; drop table got unstuck as soon as
partition is removed from metastore).
We should probably have a 2-way option for such cases - error out on load
(default), or convert to '?' (that should actually work)/drop such characters.
We should also check if partitions with '?' inserted explicitly work at all
with datanucleus.
> external table creation in Hive can create unusable partition
> -------------------------------------------------------------
>
> Key: HIVE-10722
> URL: https://issues.apache.org/jira/browse/HIVE-10722
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Priority: Critical
>
> There can be directories in HDFS containing unprintable characters; when
> doing hadoop fs -ls, these characters are not even visible, and can only be
> seen for example if output is piped thru od.
> When these are loaded, they are stored in e.g. mysql as "?" (literal question
> mark, findable via LIKE '%?%' in db) and show accordingly in Hive.
> However, datanucleus appears to encode it as %3F; this causes the partition
> to be unusable - it cannot be dropped, and other operations like drop table
> get stuck (didn't investigate in detail why; drop table got unstuck as soon
> as partition is removed from metastore).
> We should probably have a 2-way option for such cases - error out on load
> (default), or convert to '?' (that should actually work)/drop such characters.
> We should also check if partitions with '?' inserted explicitly work at all
> with datanucleus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
