Hi Wes and Animesh,
Thanks for the analysis and discussion. I am happy to looking into this. I
will create some Jiras soon.
Li
On Thu, Oct 11, 2018 at 5:49 AM Wes McKinney <wesmck...@gmail.com> wrote:
> hey Animesh,
>
> Thank you for doing this analysis. If you'd like to share some of the
> analysis more broadly e.g. on the Apache Arrow blog or social media,
> let us know.
>
> Seems like there might be a few follow ups here for the Arrow Java
> community:
>
> * Documentation about achieving better performance
> * Writing some microperformance benchmarks
> * Making some improvements to the code to facilitate better performance
>
> Feel free to create some JIRA issues. Are any Java developers
> interested in digging a little more into these issues?
>
> Thanks,
> Wes
> On Tue, Oct 9, 2018 at 7:18 AM Animesh Trivedi
> <animesh.triv...@gmail.com> wrote:
> >
> > Hi Wes and all,
> >
> > Here is another round of updates:
> >
> > Quick recap - previously we established that for 1kB binary blobs, Arrow
> > can deliver > 160 Gbps performance from in-memory buffers.
> >
> > In this round I looked at the performance of materializing "integers". In
> > my benchmarks, I found that with careful optimizations/code-rewriting we
> > can push the performance of integer reading from 5.42 Gbps/core to 13.61
> > Gbps/core (~2.5x). The peak performance with 16 cores, scale up to 110+
> > Gbps. Key things to do is:
> >
> > 1) Disable memory access checks in Arrow and Netty buffers. This gave
> > significant performance boost. However, for such an important performance
> > flag, it is very poorly documented
> > ("drill.enable_unsafe_memory_access=true").
> >
> > 2) Materialize values from Validity and Value direct buffers instead of
> > calling getInt() function on the IntVector. This is implemented as a new
> > Unsafe reader type (
> >
> https://github.com/animeshtrivedi/benchmarking-arrow/blob/master/src/main/java/com/github/animeshtrivedi/benchmark/ArrowReaderUnsafe.java#L31
> > )
> >
> > 3) Optimize bitmap operation to check if a bit is set or not (
> >
> https://github.com/animeshtrivedi/benchmarking-arrow/blob/master/src/main/java/com/github/animeshtrivedi/benchmark/ArrowReaderUnsafe.java#L23
> > )
> >
> > A detailed write up of these steps is available here:
> >
> https://github.com/animeshtrivedi/blog/blob/master/post/2018-10-09-arrow-int.md
> >
> > I have 2 follow-up questions:
> >
> > 1) Regarding the `isSet` function, why does it has to calculate number of
> > bits set? (
> >
> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java#L797
> ).
> > Wouldn't just checking if the result of the AND operation is zero or not
> be
> > sufficient? Like what I did :
> >
> https://github.com/animeshtrivedi/benchmarking-arrow/blob/master/src/main/java/com/github/animeshtrivedi/benchmark/ArrowReaderUnsafe.java#L28
> >
> >
> > 2) What is the reason behind this bitmap generation optimization here
> >
> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java#L179
> > ? At this point when this function is called, the bitmap vector is
> already
> > read from the storage, and contains the right values (either all null,
> all
> > set, or whatever). Generating this mask here for the special cases when
> the
> > values are all NULL or all set (this was the case in my benchmark), can
> be
> > slower than just returning what one has read from the storage.
> >
> > Collectively optimizing these two bitmap operations give more than 1 Gbps
> > gains in my bench-marking code.
> >
> > Cheers,
> > --
> > Animesh
> >
> >
> > On Thu, Oct 4, 2018 at 12:52 PM Wes McKinney <wesmck...@gmail.com>
> wrote:
> >
> > > See e.g.
> > >
> > >
> > >
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-write-test.cc#L222
> > >
> > >
> > > On Thu, Oct 4, 2018 at 6:48 AM Animesh Trivedi
> > > <animesh.triv...@gmail.com> wrote:
> > > >
> > > > Primarily write the same microbenchmark as I have in Java in C++ for
> > > table
> > > > reading and value materialization. So just an example of equivalent
> > > > ArrowFileReader example code in C++. Unit tests are a good starting
> > > point,
> > > > thanks for the tip :)
> > > >
> > > > On Thu, Oct 4, 2018 at 12:39 PM Wes McKinney <wesmck...@gmail.com>
> > > wrote:
> > > >
> > > > > > 3. Are there examples of Arrow in C++ read/write code that I can
> > > have a
> > > > > look?
> > > > >
> > > > > What kind of code are you looking for? I would direct you to
> relevant
> > > > > unit tests that exhibit certain functionality, but it depends on
> what
> > > > > you are trying to do
> > > > > On Wed, Oct 3, 2018 at 9:45 AM Animesh Trivedi
> > > > > <animesh.triv...@gmail.com> wrote:
> > > > > >
> > > > > > Hi all - quick update on the performance investigation:
> > > > > >
> > > > > > - I spent some time looking at performance profile for a binary
> blob
> > > > > column
> > > > > > (1024 bytes of byte[]) and found a few favorable settings for
> > > delivering
> > > > > up
> > > > > > to 168 Gbps from in-memory reading benchmark on 16 cores. These
> > > settings
> > > > > > (NUMA, JVM settings, Arrow holder API, and batch size, etc.) are
> > > > > documented
> > > > > > here:
> > > > > >
> > > > >
> > >
> https://github.com/animeshtrivedi/blog/blob/master/post/2018-10-03-arrow-binary.md
> > > > > > - these setting also help to improved the last number that
> reported
> > > (but
> > > > > > not by much) for the in-memory TPC-DS store_sales table from ~39
> > > Gbps up
> > > > > to
> > > > > > ~45-47 Gbps (note: this number is just in-memory benchmark, i.e.,
> > > w/o any
> > > > > > networking or storage links)
> > > > > >
> > > > > > A few follow up questions that I have:
> > > > > > 1. Arrow reads a batch size worth of data in one go. Are there
> any
> > > > > > recommended batch sizes? In my investigation, small batch size
> help
> > > with
> > > > > a
> > > > > > better cache profile but increase number of instructions required
> > > (more
> > > > > > looping). Larger one do otherwise. Somehow ~10MB/thread seem to
> be
> > > the
> > > > > best
> > > > > > performing configuration, which is also a bit counter intuitive
> as
> > > for 16
> > > > > > threads this will lead to 160 MB of memory footprint. May be
> this is
> > > also
> > > > > > tired to the memory management logic which is my next question.
> > > > > > 2. Arrow use's netty's memory manager. (i) what are decent netty
> > > memory
> > > > > > management settings for "io.netty.allocator.*" parameters? I
> don't
> > > find
> > > > > any
> > > > > > decent write-up on them; (ii) Is there a provision for ArrowBuf
> being
> > > > > > re-used once a batch is consumed? As it looks for now, read read
> > > > > allocates
> > > > > > a new buffer to read the whole batch size.
> > > > > > 3. Are there examples of Arrow in C++ read/write code that I can
> > > have a
> > > > > > look?
> > > > > >
> > > > > > Cheers,
> > > > > > --
> > > > > > Animesh
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 19, 2018 at 8:49 PM Wes McKinney <
> wesmck...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > On Wed, Sep 19, 2018 at 2:13 PM Animesh Trivedi
> > > > > > > <animesh.triv...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Johan, Wes, and Jacques - many thanks for your comments:
> > > > > > > >
> > > > > > > > @Johan -
> > > > > > > > 1. I also do not suspect that there is any inherent drawback
> in
> > > Java
> > > > > or
> > > > > > > C++
> > > > > > > > due to the Arrow format. I mentioned C++ because Wes pointed
> out
> > > that
> > > > > > > Java
> > > > > > > > routines are not the most optimized ones (yet!). And
> naturally
> > > one
> > > > > would
> > > > > > > > expect better performance in a native language with all
> > > > > > > pointer/memory/SIMD
> > > > > > > > instruction optimizations that you mentioned. As far as I
> know,
> > > the
> > > > > > > > off-heap buffers are managed in ArrowBuf which implements an
> > > abstract
> > > > > > > netty
> > > > > > > > class. But there is nothing unusual, i.e., netty specific,
> about
> > > > > these
> > > > > > > > unsafe routines, they are used by many projects. Though
> there is
> > > cost
> > > > > > > > associated with materializing on-heap Java values from
> off-heap
> > > > > memory
> > > > > > > > regions. I need to benchmark that more carefully.
> > > > > > > >
> > > > > > > > 2. When you say "I've so far always been able to get similar
> > > > > performance
> > > > > > > > numbers" - do you mean the same performance of my case 3
> where 16
> > > > > cores
> > > > > > > > drive close to 40 Gbps, or the same performance between your
> C++
> > > and
> > > > > Java
> > > > > > > > benchmarks. Do you have some write-up? I would be interested
> to
> > > read
> > > > > up
> > > > > > > :)
> > > > > > > >
> > > > > > > > 3. "Can you get to 100 Gbps starting from primitive arrays in
> > > Java"
> > > > > ->
> > > > > > > that
> > > > > > > > is a good idea. Let me try and report back.
> > > > > > > >
> > > > > > > > @Wes -
> > > > > > > >
> > > > > > > > Is there some benchmark template for C++ routines I can have
> a
> > > look?
> > > > > I
> > > > > > > > would be happy to get some input from Java-Arrow experts on
> how
> > > to
> > > > > write
> > > > > > > > these benchmarks more efficiently. I will have a closer look
> at
> > > the
> > > > > JIRA
> > > > > > > > tickets that you mentioned.
> > > > > > > >
> > > > > > > > So, for now I am focusing on the case 3, which is about
> > > establishing
> > > > > > > > performance when reading from a local in-memory I/O stream
> that I
> > > > > > > > implemented (
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://github.com/animeshtrivedi/benchmarking-arrow/blob/master/src/main/java/com/github/animeshtrivedi/benchmark/MemoryIOChannel.java
> > > > > > > ).
> > > > > > > > In this case I first read data from parquet files, convert
> them
> > > into
> > > > > > > Arrow,
> > > > > > > > and write-out to a MemoryIOChannel, and then read back from
> it.
> > > So,
> > > > > the
> > > > > > > > performance has nothing to do with Crail or HDFS in the case
> 3.
> > > > > Once, I
> > > > > > > > establish the base performance in this setup (which is
> around ~40
> > > > > Gbps
> > > > > > > with
> > > > > > > > 16 cores) I will add Crail to the mix. Perhaps Crail I/O
> streams
> > > can
> > > > > take
> > > > > > > > ArrowBuf as src/dst buffers. That should be doable.
> > > > > > >
> > > > > > > Right, in any case what you are testing is the performance of
> > > using a
> > > > > > > particular Java accessor layer to JVM off-heap Arrow memory to
> sum
> > > the
> > > > > > > non-null values of each column. I'm not sure that a single
> > > bandwidth
> > > > > > > number produced by this benchmark is very informative for
> people
> > > > > > > contemplating what memory format to use in their system due to
> the
> > > > > > > current state of the implementation (Java) and workload
> measured
> > > > > > > (summing the non-null values with a naive algorithm). I would
> guess
> > > > > > > that a C++ version with raw pointers and a loop-unrolled,
> > > branch-free
> > > > > > > vectorized sum is going to be a lot faster.
> > > > > > >
> > > > > > > >
> > > > > > > > @Jacques -
> > > > > > > >
> > > > > > > > That is a good point that "Arrow's implementation is more
> > > focused on
> > > > > > > > interacting with the structure than transporting it".
> However,
> > > in any
> > > > > > > > distributed system one needs to move data/structure around -
> as
> > > far
> > > > > as I
> > > > > > > > understand that is another goal of the project. My
> investigation
> > > > > started
> > > > > > > > within the context of Spark/SQL data processing. Spark
> converts
> > > > > incoming
> > > > > > > > data into its own in-memory UnsafeRow representation for
> > > processing.
> > > > > So
> > > > > > > > naturally the performance of this data ingestion pipeline
> cannot
> > > > > > > outperform
> > > > > > > > the read performance of the used file format. I benchmarked
> > > Parquet,
> > > > > ORC,
> > > > > > > > Avro, JSON (for the specific TPC-DS store_sales table). And
> then
> > > > > > > curiously
> > > > > > > > benchmarked Arrow as well because its design choices are a
> better
> > > > > fit for
> > > > > > > > modern high-performance RDMA/NVMe/100+Gbps devices I am
> > > > > investigating.
> > > > > > > From
> > > > > > > > this point of view, I am trying to find out can Arrow be the
> file
> > > > > format
> > > > > > > > for the next generation of storage/networking devices (see
> Apache
> > > > > Crail
> > > > > > > > project) delivering close to the hardware speed
> reading/writing
> > > > > rates. As
> > > > > > > > Wes pointed out that a C++ library implementation should be
> > > memory-IO
> > > > > > > > bound, so what would it take to deliver the same performance
> in
> > > Java
> > > > > ;)
> > > > > > > > (and then, from across the network).
> > > > > > > >
> > > > > > > > I hope this makes sense.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > --
> > > > > > > > Animesh
> > > > > > > >
> > > > > > > > On Wed, Sep 19, 2018 at 6:28 PM Jacques Nadeau <
> > > jacq...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > My big question is what is the use case and how/what are
> you
> > > > > trying to
> > > > > > > > > compare? Arrow's implementation is more focused on
> interacting
> > > > > with the
> > > > > > > > > structure than transporting it. Generally speaking, when
> we're
> > > > > working
> > > > > > > with
> > > > > > > > > Arrow data we frequently are just interacting with memory
> > > > > locations and
> > > > > > > > > doing direct operations. If you have a layer that supports
> that
> > > > > type of
> > > > > > > > > semantic, create a movement technique that depends on that.
> > > Arrow
> > > > > > > doesn't
> > > > > > > > > force a particular API since the data itself is defined by
> its
> > > > > > > in-memory
> > > > > > > > > layout so if you have a custom use or pattern, just work
> with
> > > the
> > > > > > > in-memory
> > > > > > > > > structures.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Sep 19, 2018 at 7:49 AM Wes McKinney <
> > > wesmck...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > hi Animesh,
> > > > > > > > > >
> > > > > > > > > > Per Johan's comments, the C++ library is essentially
> going
> > > to be
> > > > > > > > > > IO/memory bandwidth bound since you're interacting with
> raw
> > > > > pointers.
> > > > > > > > > >
> > > > > > > > > > I'm looking at your code
> > > > > > > > > >
> > > > > > > > > > private void consumeFloat4(FieldVector fv) {
> > > > > > > > > > Float4Vector accessor = (Float4Vector) fv;
> > > > > > > > > > int valCount = accessor.getValueCount();
> > > > > > > > > > for(int i = 0; i < valCount; i++){
> > > > > > > > > > if(!accessor.isNull(i)){
> > > > > > > > > > float4Count+=1;
> > > > > > > > > > checksum+=accessor.get(i);
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > You'll want to get a Java-Arrow expert from Dremio to
> advise
> > > you
> > > > > the
> > > > > > > > > > fastest way to iterate over this data -- my
> understanding is
> > > that
> > > > > > > much
> > > > > > > > > > code in Dremio interacts with the wrapped Netty ArrowBuf
> > > objects
> > > > > > > > > > rather than going through the higher level APIs. You're
> also
> > > > > dropping
> > > > > > > > > > performance because memory mapping is not yet
> implemented in
> > > > > Java,
> > > > > > > see
> > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-3191.
> > > > > > > > > >
> > > > > > > > > > Furthermore, the IPC reader class you are using could be
> made
> > > > > more
> > > > > > > > > > efficient. I described the problem in
> > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-3192 -- this
> > > will be
> > > > > > > > > > required as soon as we have the ability to do memory
> mapping
> > > in
> > > > > Java
> > > > > > > > > >
> > > > > > > > > > Could Crail use the Arrow data structures in its runtime
> > > rather
> > > > > than
> > > > > > > > > > copying? If not, how are Crail's runtime data structures
> > > > > different?
> > > > > > > > > >
> > > > > > > > > > - Wes
> > > > > > > > > > On Wed, Sep 19, 2018 at 9:19 AM Johan Peltenburg - EWI
> > > > > > > > > > <j.w.peltenb...@tudelft.nl> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hello Animesh,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I browsed a bit in your sources, thanks for sharing. We
> > > have
> > > > > > > performed
> > > > > > > > > > some similar measurements to your third case in the past
> for
> > > > > C/C++ on
> > > > > > > > > > collections of various basic types such as primitives and
> > > > > strings.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I can say that in terms of consuming data from the
> Arrow
> > > format
> > > > > > > versus
> > > > > > > > > > language native collections in C++, I've so far always
> been
> > > able
> > > > > to
> > > > > > > get
> > > > > > > > > > similar performance numbers (e.g. no drawbacks due to the
> > > Arrow
> > > > > > > format
> > > > > > > > > > itself). Especially when accessing the data through
> Arrow's
> > > raw
> > > > > data
> > > > > > > > > > pointers (and using for example std::string_view-like
> > > > > constructs).
> > > > > > > > > > >
> > > > > > > > > > > In C/C++ the fast data structures are engineered in
> such a
> > > way
> > > > > > > that as
> > > > > > > > > > little pointer traversals are required and they take up
> an as
> > > > > small
> > > > > > > as
> > > > > > > > > > possible memory footprint. Thus each memory access is
> > > relatively
> > > > > > > > > efficient
> > > > > > > > > > (in terms of obtaining the data of interest). The same
> can
> > > > > > > absolutely be
> > > > > > > > > > said for Arrow, if not even more efficient in some cases
> > > where
> > > > > object
> > > > > > > > > > fields are of variable length.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > In the JVM case, the Arrow data is stored off-heap.
> This
> > > > > requires
> > > > > > > the
> > > > > > > > > > JVM to interface to it through some calls to Unsafe
> hidden
> > > under
> > > > > the
> > > > > > > > > Netty
> > > > > > > > > > layer (but please correct me if I'm wrong, I'm not an
> expert
> > > on
> > > > > > > this).
> > > > > > > > > > Those calls are the only reason I can think of that would
> > > > > degrade the
> > > > > > > > > > performance a bit compared to a pure JAva case. I don't
> know
> > > if
> > > > > the
> > > > > > > > > Unsafe
> > > > > > > > > > calls are inlined during JIT compilation. If they aren't
> they
> > > > > will
> > > > > > > > > increase
> > > > > > > > > > access latency to any data a little bit.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I don't have a similar machine so it's not easy to
> relate
> > > my
> > > > > > > numbers to
> > > > > > > > > > yours, but if you can get that data consumed with 100
> Gbps
> > > in a
> > > > > pure
> > > > > > > Java
> > > > > > > > > > case, I don't see any reason (resulting from Arrow
> format /
> > > > > off-heap
> > > > > > > > > > storage) why you wouldn't be able to get at least really
> > > close.
> > > > > Can
> > > > > > > you
> > > > > > > > > get
> > > > > > > > > > to 100 Gbps starting from primitive arrays in Java with
> your
> > > > > > > consumption
> > > > > > > > > > functions in the first place?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I'm interested to see your progress on this.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Kind regards,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Johan Peltenburg
> > > > > > > > > > >
> > > > > > > > > > > ________________________________
> > > > > > > > > > > From: Animesh Trivedi <animesh.triv...@gmail.com>
> > > > > > > > > > > Sent: Wednesday, September 19, 2018 2:08:50 PM
> > > > > > > > > > > To: dev@arrow.apache.org; d...@crail.apache.org
> > > > > > > > > > > Subject: [JAVA] Arrow performance measurement
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > A week ago, Wes and I had a discussion about the
> > > performance
> > > > > of the
> > > > > > > > > > > Arrow/Java implementation on the Apache Crail
> (Incubating)
> > > > > mailing
> > > > > > > > > list (
> > > > > > > > > > >
> > > > > > >
> > > http://mail-archives.apache.org/mod_mbox/crail-dev/201809.mbox/browser
> > > > > > > > > ).
> > > > > > > > > > In
> > > > > > > > > > > a nutshell: I am investigating the performance of
> various
> > > file
> > > > > > > formats
> > > > > > > > > > > (including Arrow) on high-performance NVMe and
> > > > > RDMA/100Gbps/RoCE
> > > > > > > > > setups.
> > > > > > > > > > I
> > > > > > > > > > > benchmarked how long does it take to materialize values
> > > (ints,
> > > > > > > longs,
> > > > > > > > > > > doubles) of the store_sales table, the largest table
> in the
> > > > > TPC-DS
> > > > > > > > > > dataset
> > > > > > > > > > > stored on different file formats. Here is a write-up on
> > > this -
> > > > > > > > > > >
> > > https://crail.incubator.apache.org/blog/2018/08/sql-p1.html. I
> > > > > > > found
> > > > > > > > > > that
> > > > > > > > > > > between a pair of machine connected over a 100 Gbps
> link,
> > > Arrow
> > > > > > > (using
> > > > > > > > > > as a
> > > > > > > > > > > file format on HDFS) delivered close to ~30 Gbps of
> > > bandwidth
> > > > > with
> > > > > > > all
> > > > > > > > > 16
> > > > > > > > > > > cores engaged. Wes pointed out that (i) Arrow is
> in-memory
> > > IPC
> > > > > > > format,
> > > > > > > > > > and
> > > > > > > > > > > has not been optimized for storage interfaces/APIs like
> > > HDFS;
> > > > > (ii)
> > > > > > > the
> > > > > > > > > > > performance I am measuring is for the java
> implementation.
> > > > > > > > > > >
> > > > > > > > > > > Wes, I hope I summarized our discussion correctly.
> > > > > > > > > > >
> > > > > > > > > > > That brings us to this email where I promised to
> follow up
> > > on
> > > > > the
> > > > > > > Arrow
> > > > > > > > > > > mailing list to understand and optimize the
> performance of
> > > > > > > Arrow/Java
> > > > > > > > > > > implementation on high-performance devices. I wrote a
> small
> > > > > > > stand-alone
> > > > > > > > > > > benchmark (
> > > > > https://github.com/animeshtrivedi/benchmarking-arrow)
> > > > > > > with
> > > > > > > > > > three
> > > > > > > > > > > implementations of WritableByteChannel,
> SeekableByteChannel
> > > > > > > interfaces:
> > > > > > > > > > >
> > > > > > > > > > > 1. Arrow data is stored in HDFS/tmpfs - this gives me
> ~30
> > > Gbps
> > > > > > > > > > performance
> > > > > > > > > > > 2. Arrow data is stored in Crail/DRAM - this gives me
> > > ~35-36
> > > > > Gbps
> > > > > > > > > > > performance
> > > > > > > > > > > 3. Arrow data is stored in on-heap byte[] - this gives
> me
> > > ~39
> > > > > Gbps
> > > > > > > > > > > performance
> > > > > > > > > > >
> > > > > > > > > > > I think the order makes sense. To better understand the
> > > > > > > performance of
> > > > > > > > > > > Arrow/Java we can focus on the option 3.
> > > > > > > > > > >
> > > > > > > > > > > The key question I am trying to answer is "what would
> it
> > > take
> > > > > for
> > > > > > > > > > > Arrow/Java to deliver 100+ Gbps of performance"? Is it
> > > > > possible? If
> > > > > > > > > yes,
> > > > > > > > > > > then what is missing/or mis-interpreted by me? If not,
> then
> > > > > where
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > > performance lost? Does anyone have any performance
> > > measurements
> > > > > > > for C++
> > > > > > > > > > > implementation? if they have seen/expect better
> numbers.
> > > > > > > > > > >
> > > > > > > > > > > As a next step, I will profile the read path of
> Arrow/Java
> > > for
> > > > > the
> > > > > > > > > option
> > > > > > > > > > > 3. I will report my findings.
> > > > > > > > > > >
> > > > > > > > > > > Any thoughts and feedback on this investigation are
> very
> > > > > welcome.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > --
> > > > > > > > > > > Animesh
> > > > > > > > > > >
> > > > > > > > > > > PS~ Cross-posting on the d...@crail.apache.org list as
> > > well.
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
