dongwoo.kim created FLINK-32760:
-----------------------------------
Summary: Version Conflict in flink-sql-connector-hive for
shaded.parquet prefix packages
Key: FLINK-32760
URL: https://issues.apache.org/jira/browse/FLINK-32760
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Reporter: dongwoo.kim
Attachments: image-2023-08-05-14-50-47-806.png
h2. Summary
In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading
parquet dependency from *hive-exec* is done.
But I think this is not enough and causing errors like below when I try to read
parquet file using sql-gateway which requires both *flink-parquet* and
*flink-sql-connector-hive* dependencies.
!image-2023-08-05-14-50-47-806.png|width=1392,height=909!
h2. {color:#172b4d}Cause{color}
{color:#172b4d}Parquet dependency not only includes org.apache.parquet but also
shaded.parquet prefix dependencies.
([ref|https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/pom.xml#L72]
{color}
{color:#172b4d}- flink-parquet depends on Parquet 1.12.3 with shaded Thrift
0.16.0 (prefix: {{{}shaded.parquet{}}}){color}
{color:#172b4d}- flink-sql-connector-hive depends on hive-exec 3.1.3 with
Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: {{{}shaded.parquet{}}}){color}
{color:#172b4d}- Code compiled against Thrift 0.16.0 attempts to run against
0.9.3, causing the error.{color}
h2. {color:#172b4d}Proposed solution{color}
Adding new shading rule to flink-sql-connector-hive project.
I have confirmed that if we add below rule, the above error is resolved.
{code:xml}
<relocation>
<pattern>shaded.parquet</pattern>
<shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern>
</relocation>{code}
I would be happy to implement it if the proposal is accepted. Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
