> On Feb. 7, 2017, 2:31 a.m., Owen O'Malley wrote: > > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java, line 360 > > <https://reviews.apache.org/r/55977/diff/6/?file=1625756#file1625756line360> > > > > You need braces around your control block for break. > > > > Actually, this is an error to have an include vector that is shorter > > that the reader type and should cause a thrown exception. Although it > > should be a single check that include.length == readerSchema.getMaxId() + 1. > > Sergey Shelukhin wrote: > as far as I remember, I've seen vectorization(?) use trimmed includes > somewhere > > Owen O'Malley wrote: > Nope. The include vector has to exactly match the reader schema, although > it doesn't include the ACID columns. In fact, the current version of > SchemaEvolution in ORC will throw in such cases. > > Sergey Shelukhin wrote: > See e.g. VectorMapOperator - determineDataColumnsToIncludeTruncated > > Owen O'Malley wrote: > If that boolean array is passed down to an ORC reader (or directly to > SchemaEvolution), it will fail by design.
dataColumnsToIncludeTruncated is designed to be used with Text (LazySimpleDeserializeRead class) to minimized the number of text fields that have to be parsed. From a recommendation by Gopal who observed a bunch of text fields being parsed that didn't need to be. Schema Evolution does work with vectorized text because the Vectorizer class currently insists the partition names match the table schema (a restricted positional requirement). dataColumnsToIncludeTruncated is not used with ORC. - Matt ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55977/#review164437 ----------------------------------------------------------- On Feb. 8, 2017, 1:10 a.m., Sergey Shelukhin wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55977/ > ----------------------------------------------------------- > > (Updated Feb. 8, 2017, 1:10 a.m.) > > > Review request for hive, Gopal V and Prasanth_J. > > > Repository: hive-git > > > Description > ------- > > see jira > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f3b01b2 > > llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java > 85fae9a > > llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java > 4295c1c > > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java > 8d86d17 > > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VertorDeserializeOrcWriter.java > 63a3be2 > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 99cc506 > > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java > a434763 > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 1c5f0e6 > > storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java > b894c11e > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java > bbd9ca6 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java > 6f090a1 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/DecimalColumnVector.java > e4f8d82 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java > bd421f4 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/IntervalDayTimeColumnVector.java > c4a6c0f > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java > 80d4731 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/MultiValuedColumnVector.java > 1aeff83 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/StructColumnVector.java > cf07bca > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/TimestampColumnVector.java > 28997a0 > > storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/UnionColumnVector.java > 0c61243 > > Diff: https://reviews.apache.org/r/55977/diff/ > > > Testing > ------- > > > Thanks, > > Sergey Shelukhin > >
