[
https://issues.apache.org/jira/browse/IMPALA-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955582#comment-17955582
]
ASF subversion and git services commented on IMPALA-10319:
----------------------------------------------------------
Commit 4837cedc795017aa1e7b69ef0914020a3022ca88 in impala's branch
refs/heads/master from Mihaly Szjatinya
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4837cedc7 ]
IMPALA-10319: Support arbitrary encodings on Text files
As proposed in Jira, this implements decoding and encoding of text
buffers for Impala/Hive text tables. Given a table with
'serialization.encoding' property set, similarly to Hive, Impala should
be able to encode the inserted data into charset specified, consequently
saving it into a text file. The opposite decoding operation should be
performed upon reading data buffers from text files. Both operations
employ boost::locale::conv library.
Since Hive doesn't encode line delimiters, charsets that would have
delimiters stored differently from ASCII are not allowed.
One difference from Hive is that Impala implements
'serialization.encoding' only as a per partition serdeproperty to avoid
confusion of allowing both serde and tbl properties. (See related
IMPALA-13748)
Note: Due to precreated non-UTF-8 files present in the patch
'gerrit-code-review-checks' was performed locally. (See IMPALA-14100)
Change-Id: I787cd01caa52a19d6645519a6cedabe0a5253a65
Reviewed-on: http://gerrit.cloudera.org:8080/22049
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Support arbitrary encodings on Text files
> -----------------------------------------
>
> Key: IMPALA-10319
> URL: https://issues.apache.org/jira/browse/IMPALA-10319
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Mihaly Szjatinya
> Priority: Critical
> Attachments: gbk_names.txt
>
>
> ORC/Parquet/Avro files store strings in UTF-8 encoded bytes. However, Text
> and Sequence files can be in arbitrary encodings. Hive supports specifying
> arbitrary encoding on tables using LazySimpleSerDe with the
> "serialization.encoding" table property (HIVE-7142). Impala is currently not
> aware of this table property and treate all strings as byte arrays. It's good
> to support at least reading from these text/sequence files.
> *Example*
> Create a text table in Hive using GBK encoding and load a GBK encoded text
> file into it:
> {code:sql}
> hive> create table gbk_names (name string) stored as textfile
> tblproperties("serialization.encoding"="GBK");
> hive> load data local inpath '/home/quanlong/workspace/Impala/gbk_names.txt'
> into table gbk_names;
> hive> select * from gbk_names;
> +-----------------+
> | gbk_names.name |
> +-----------------+
> | 张三 |
> | 李四 |
> | 王五 |
> +-----------------+
> {code}
> Impala read strings as byte arrays so can't decode them correctly:
> {code:sql}
> impala-shell> invalidate metadata gbk_names;
> impala-shell> select * from gbk_names;
> +------+
> | name |
> +------+
> | ���� |
> | ���� |
> | ���� |
> +------+
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
