xsys created HIVE-26531: --------------------------- Summary: UnsupportedOperationException while creating table in Avro format if column schema contains MAP with INTEGER key Key: HIVE-26531 URL: https://issues.apache.org/jira/browse/HIVE-26531 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 3.1.2 Reporter: xsys Attachments: Avro_Map_StackTrace.txt
h3. Describe the bug We are trying to save a table with the {{Avro}} data format through {{{}spark-sql{}}}. The table contains {{MAP}} as part of the schema and the map's key is an {{{}INT{}}}: {{{}MAP<INT, STRING>{}}}. We observe the following exception from the {{CREATE TABLE}} query: {noformat} 22/08/29 12:03:38 ERROR Table: Unable to get field from serde: org.apache.hadoop.hive.serde2.avro.AvroSerDe java.lang.UnsupportedOperationException: Key of Map can only be a String{noformat} _Here is the full stack trace, for reference:_ [^Avro_Map_StackTrace.txt] The exception is raised by the following [Hive code|https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java#L216-L221]: {noformat} private Schema createAvroMap(TypeInfo typeInfo) { TypeInfo keyTypeInfo = ((MapTypeInfo) typeInfo).getMapKeyTypeInfo(); if (((PrimitiveTypeInfo) keyTypeInfo).getPrimitiveCategory() != PrimitiveObjectInspector.PrimitiveCategory.STRING) { throw new UnsupportedOperationException("Key of Map can only be a String"); } TypeInfo valueTypeInfo = ((MapTypeInfo) typeInfo).getMapValueTypeInfo(); Schema valueSchema = createAvroSchema(valueTypeInfo); return Schema.createMap(valueSchema); }{noformat} h3. To Reproduce On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{spark-shell}} with the Avro package: {noformat} $SPARK_HOME/bin/spark-sql --packages org.apache.spark:spark-avro_2.12:3.2.1 {noformat} Execute the following: {noformat} create table avro_map(c1 MAP<INT, STRING>) ROW FORMAT SERDE "org.apache.hadoop.hive.serde2.avro.AvroSerDe" STORED AS INPUTFORMAT "org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat" OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat";{noformat} h3. Expected behavior We expect to create a table successfully in Avro format if the column schema contains MAP with INTEGER key. We tried other formats like Parquet & ORC and the outcome is consistent with this expectation. Here is a simplified example showing expected behavior using the Parquet & ORC file formats: {noformat} spark-sql> create table orc_map(c1 MAP<INT, STRING>) STORED AS ORC; Time taken: 0.196 seconds spark-sql> create table parquet_map(c1 MAP<INT, STRING>) STORED AS PARQUET; Time taken: 0.113 seconds spark-sql> desc orc_map; c1 map<int,string> Time taken: 0.387 seconds, Fetched 1 row(s) spark-sql> desc parquet_map; c1 map<int,string> Time taken: 0.077 seconds, Fetched 1 row(s){noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)