zenfenan commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink URL: https://github.com/apache/flink/pull/11474#discussion_r402049864
########## File path: flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/vectorizer/Vectorizer.java ########## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.formats.orc.vectorizer; + +import org.apache.flink.annotation.PublicEvolving; + +import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch; + +import java.io.IOException; +import java.io.Serializable; + +/** + * Implementors of this interface provide the logic to transform their data to {@link VectorizedRowBatch}. + * + * @param <T> The type of the element + */ +@PublicEvolving +public interface Vectorizer<T> extends Serializable { + + /** + * Creates a VectorizedRowBatch containing an array of ColumnVectors + * from the provided element. + * + * @param element The input element + * @return The VectorizedRowBatch containing the ColumnVectors of the input element + * @throws IOException if there is an error while transforming the input. + */ + VectorizedRowBatch vectorize(T element) throws IOException; Review comment: @JingsongLi Earlier As @kl0u had mentioned, regardless of being a single element VectorizedRowBatch, they end up in the same stripe. TBH, I tested two approaches initially when I working on this: 1. Vectorizer interface has two methods `addElements(T element)` which just buffers the element in a list and another method `vectorize()` which just iterates and added them into a single `VectorizedRowBatch`. 2. Vectorizer interface has one method `vectorize(T element)` which adds that element alone into a `VectorizedRowBatch` Upon testing, I found there was no size difference between the two approaches, so I avoided the complexity of additional buffering and went approach 2. Having said that, just to clarify, I will be reaching out to the ORC community to understand the difference and see if we are losing any efficiencies in this approach. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
