Wes McKinney created ARROW-2800:
-----------------------------------
Summary: [Python] Unavailable Parquet column statistics from
Spark-generated file
Key: ARROW-2800
URL: https://issues.apache.org/jira/browse/ARROW-2800
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.9.0
Reporter: Robert Gruener
Fix For: 0.10.0
I have a dataset generated by spark which shows it has statistics for the
string column when using the java parquet-mr code (shown by using
`parquet-tools meta`) however reading from pyarrow shows that the statistics
for that column are not set. I should not the column only has a single value,
though it still seems like a problem that pyarrow can't recognize it (it can
recognize statistics set for the long and double types).
See https://github.com/apache/arrow/files/2161147/metadata.zip for file example.
Pyarrow Code To Check Statistics:
{code}
from pyarrow import parquet as pq
meta = pq.read_metadata('/tmp/metadata.parquet')
# No Statistics For String Column, prints false and statistics object is None
print(meta.row_group(0).column(1).is_stats_set)
{code}
Example parquet-meta output:
{code}
file schema: spark_schema
--------------------------------------------------------------------------------
int: REQUIRED INT64 R:0 D:0
string: OPTIONAL BINARY O:UTF8 R:0 D:1
float: REQUIRED DOUBLE R:0 D:0
row group 1: RC:8333 TS:76031 OFFSET:4
--------------------------------------------------------------------------------
int: INT64 SNAPPY DO:0 FPO:4 SZ:7793/8181/1.05 VC:8333
ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 100, num_nulls: 0]
string: BINARY SNAPPY DO:0 FPO:7797 SZ:1146/1139/0.99 VC:8333
ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE ST:[min: hello, max: hello, num_nulls: 4192]
float: DOUBLE SNAPPY DO:0 FPO:8943 SZ:66720/66711/1.00 VC:8333
ENC:PLAIN,BIT_PACKED ST:[min: 0.0057611096964338415, max: 99.99811053829232,
num_nulls: 0]
{code}
I realize the column only has a single value though it still seems like pyarrow
should be able to read the statistics set. I made this here and not a JIRA
since I wanted to be sure this is actually an issue and there wasnt a ticket
already made there (I couldnt find one but I wanted to be sure). Either way I
would like to understand why this is
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
