[
https://issues.apache.org/jira/browse/SPARK-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen reopened SPARK-18860:
-------------------------------
> Update Parquet to 1.9.0
> -----------------------
>
> Key: SPARK-18860
> URL: https://issues.apache.org/jira/browse/SPARK-18860
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Dongjoon Hyun
>
> This issue aims to update Parquet to 1.9.0 and remove the hacks due to
> Parquet 1.8.1 limitation.
> {code}
> - // !! HACK ALERT !!
> - //
> - // PARQUET-363 & PARQUET-278: parquet-mr 1.8.1 doesn't allow constructing
> empty GroupType,
> - // which prevents us to avoid selecting any columns for queries like
> `SELECT COUNT(*) FROM t`.
> - // This issue has been fixed in parquet-mr 1.8.2-SNAPSHOT.
> - //
> - // To workaround this problem, here we first construct a `MessageType`
> with a single dummy
> - // field, and then remove the field to obtain an empty `MessageType`.
> - //
> - // TODO Reverts this change after upgrading parquet-mr to 1.8.2+
> val EMPTY_MESSAGE = Types
> .buildMessage()
> - .required(PrimitiveType.PrimitiveTypeName.INT32).named("dummy")
> .named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
> - EMPTY_MESSAGE.getFields.clear()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
