[ 
https://issues.apache.org/jira/browse/ORC-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469604#comment-17469604
 ] 

Bobby Wang commented on ORC-1075:
---------------------------------

Even `index.getNumberOfValues()` can return the actual number, but because 
there is no ColumnStatistic in RowIndex, so 

the index will not match any Integer/Double ... ColumnStatistics, finally it 
goes into 
{code:java}
new ValueRange(predicate, null, null, true);{code}
and get the same result.

 
{code:java}
// 1.8.0-SNAPSHOT
static ValueRange getValueRange(ColumnStatistics index,
                                PredicateLeaf predicate,
                                boolean useUTCTimestamp) {
  if (index.getNumberOfValues() == 0) {
    return new ValueRange<>(predicate, index.hasNull());
  } else if (index instanceof IntegerColumnStatistics) {
    IntegerColumnStatistics stats = (IntegerColumnStatistics) index;
    Long min = stats.getMinimum();
    Long max = stats.getMaximum();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else if (index instanceof CollectionColumnStatistics) {
    CollectionColumnStatistics stats = (CollectionColumnStatistics) index;
    Long min = stats.getMinimumChildren();
    Long max = stats.getMaximumChildren();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  }else if (index instanceof DoubleColumnStatistics) {
    DoubleColumnStatistics stats = (DoubleColumnStatistics) index;
    Double min = stats.getMinimum();
    Double max = stats.getMaximum();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else if (index instanceof StringColumnStatistics) {
    StringColumnStatistics stats = (StringColumnStatistics) index;
    return new ValueRange<>(predicate, stats.getLowerBound(),
        stats.getUpperBound(), stats.hasNull(), stats.getMinimum() == null,
        stats.getMaximum() == null);
  } else if (index instanceof DateColumnStatistics) {
    DateColumnStatistics stats = (DateColumnStatistics) index;
    ChronoLocalDate min = stats.getMinimumLocalDate();
    ChronoLocalDate max = stats.getMaximumLocalDate();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else if (index instanceof DecimalColumnStatistics) {
    DecimalColumnStatistics stats = (DecimalColumnStatistics) index;
    HiveDecimal min = stats.getMinimum();
    HiveDecimal max = stats.getMaximum();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else if (index instanceof TimestampColumnStatistics) {
    TimestampColumnStatistics stats = (TimestampColumnStatistics) index;
    Timestamp min = useUTCTimestamp ? stats.getMinimumUTC() : 
stats.getMinimum();
    Timestamp max = useUTCTimestamp ? stats.getMaximumUTC() : 
stats.getMaximum();
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else if (index instanceof BooleanColumnStatistics) {
    BooleanColumnStatistics stats = (BooleanColumnStatistics) index;
    Boolean min = stats.getFalseCount() == 0;
    Boolean max = stats.getTrueCount() != 0;
    return new ValueRange<>(predicate, min, max, stats.hasNull());
  } else {
    return new ValueRange(predicate, null, null, true);
  }
}
 {code}

> Failed to read rows from the ORC file without statistics in RowIndex when 
> filter is pushed down for 1.6.11
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: ORC-1075
>                 URL: https://issues.apache.org/jira/browse/ORC-1075
>             Project: ORC
>          Issue Type: Bug
>          Components: Java, Reader
>    Affects Versions: 1.6.11
>            Reporter: Bobby Wang
>            Priority: Blocker
>         Attachments: none-1.orc
>
>
> I have attached an ORC file that seems not to include ColumnStatistics in 
> RowIndex.
> {color:#FF0000}From the ORC spec, seems RowIndex.ColumnStatistics is not a 
> required field ???{color}
>  
> {code:java}
> message RowIndexEntry {
>   repeated uint64 positions = 1 [packed=true];
>   optional ColumnStatistics statistics = 2;
> }
> message RowIndex {
>   repeated RowIndexEntry entry = 1;                                           
>              
> }
> {code}
> The meta of the ORC file
>  
> {code:java}
> $ orctools meta none.orc 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.util.Shell).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Processing data file none.orc [length: 124]
> Structure for none.orc
> File Version: 0.12 with ORIGINAL
> Rows: 3
> Compression: NONE
> Calendar: Julian/Gregorian
> Type: struct<INT:int>
> Stripe Statistics:
>   Stripe 1:
>     Column 0: count: 3 hasNull: true
>     Column 1: count: 3 hasNull: true min: 1 max: 3 sum: 6
> File Statistics:
> Stripes:
>   Stripe: offset: 3 data: 4 rows: 3 tail: 32 index: 10
>     Stream: column 0 section ROW_INDEX start: 3 length 4
>     Stream: column 1 section ROW_INDEX start: 7 length 6
>     Stream: column 1 section DATA start: 13 length 4
>     Encoding column 0: DIRECT
>     Encoding column 1: DIRECT_V2
> File length: 124 bytes
> Padding length: 0 bytes
> Padding ratio: 0%
> {code}
>  
> the data of the orc file
> {code:java}
> $ orctools data none.orc 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.util.Shell).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Processing data file none.orc [length: 124]
> {"INT":1}
> {"INT":2}
> {"INT":3}{code}
> I have below code trying to read each row of the ORC file
> {code:java}
> // Pick the schema we want to read using schema evolution
> TypeDescription readSchema =
> TypeDescription.fromString("struct<INT:int>");
> // Get the information from the file footer
> Reader reader = OrcFile.createReader(new Path("none.orc"),
>                 OrcFile.readerOptions(new Configuration()));
> System.out.println("File schema: " + reader.getSchema());
> System.out.println("Row count: " + reader.getNumberOfRows());
> RecordReader rowIterator = reader.rows(
>  reader.options()
>      .schema(readSchema)
>      .searchArgument(SearchArgumentFactory.newBuilder()
>          .equals("INT", PredicateLeaf.Type.LONG, 2L)
>      .build(), new String[]{"INT"}) //predict push down
> );
> // Read the row data
> VectorizedRowBatch batch = readSchema.createRowBatch();
> LongColumnVector x = (LongColumnVector) batch.cols[0];
> while (rowIterator.nextBatch(batch)) {
>   System.out.println(batch.size);
>   for (int row = 0; row < batch.size; ++row) {
>     int xRow = x.isRepeating ? 0 : row;
>     System.out.println("INT: " + (x.noNulls || !x.isNull[xRow] ?    
>                   x.vector[xRow] :null));
>   }
> }
> rowIterator.close();{code}
>  
> h2. output from 1.6.11
> File schema: struct<INT:int>
> Row count: 3
> h2. output from 1.5.10
> File schema: struct<INT:int>
> Row count: 3
> 3
> INT: 1
> INT: 2
> INT: 3
>  
> Actually, I found this issue on Spark 3.2 which depends on ORC 1.6.11, while 
> there is no such issue on spark 3.0.x which depends on ORC 1.5.10
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to