). If you opt for this, I would strongly suggest start the
> >>> discussion on the mailing-list in order to coordinate with other
> >>> developers.
> >>>
> >>> Best regards
> >>>
> >>> Antoine.
> >>>
> >&g
rit :
>>>> Hi Antoine,
>>>>
>>>>> The question, though, is: do you *need* those higher speeds on
>> localhost?
>>>>> In which context are you considering Flight?
>>>>
>>>> We want to send large data(in cache) to th
t; In which context are you considering Flight?
> >>
> >> We want to send large data(in cache) to the data analytic application(in
> >> local).
> >>
> >> Thanks,
> >> Jiajia
> >>
> >> -Original Message-
> >&g
st?
>>> In which context are you considering Flight?
>>
>> We want to send large data(in cache) to the data analytic application(in
>> local).
>>
>> Thanks,
>> Jiajia
>>
>> -Original Message-
>> From: Antoine Pitrou
>>
Message-
> From: Antoine Pitrou
> Sent: Saturday, April 25, 2020 1:01 AM
> To: dev@arrow.apache.org
> Subject: Re: Question regarding Arrow Flight Throughput
>
>
> Hi Jiajia,
>
> It's true one should be able to reach higher speeds. For example, I can
Sent: Saturday, April 25, 2020 1:01 AM
To: dev@arrow.apache.org
Subject: Re: Question regarding Arrow Flight Throughput
Hi Jiajia,
It's true one should be able to reach higher speeds. For example, I can reach
more than 7 GB/s on a simple TCP connection, in pure Python, using only two
thr
ottleneck when running benchmark,
> I think flight can get a higher throughput.
>
> Thanks,
> Jiajia
>
> -Original Message-
> From: Antoine Pitrou
> Sent: Friday, April 24, 2020 5:47 PM
> To: dev@arrow.apache.org
> Subject: Re: Question regarding Arrow Flight Thr
riday, April 24, 2020 5:47 PM
To: dev@arrow.apache.org
Subject: Re: Question regarding Arrow Flight Throughput
The problem with gRPC is that it was designed with relatively small requests
and payloads in mind. We're using it for a large data application which it
wasn't optimized for. Also
Hi Wes,
Thanks for your reply!
Thanks,
Jiajia
-Original Message-
From: Wes McKinney
Sent: Friday, April 24, 2020 11:15 AM
To: dev
Subject: Re: Question regarding Arrow Flight Throughput
On Thu, Apr 23, 2020 at 10:02 PM Wes McKinney wrote:
>
> hi Jiajia,
>
> See
The problem with gRPC is that it was designed with relatively small
requests and payloads in mind. We're using it for a large data
application which it wasn't optimized for. Also, its threading model is
inscrutable (yielding those weird benchmark results).
However, 5 GB/s is indeed very good i
On Thu, Apr 23, 2020 at 10:02 PM Wes McKinney wrote:
>
> hi Jiajia,
>
> See my TODO here
>
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc#L182
>
> My guess is that if you want to get faster throughput with multiple
> cores, you need to run more than one serv
hi Jiajia,
See my TODO here
https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc#L182
My guess is that if you want to get faster throughput with multiple
cores, you need to run more than one server and serve on different
ports rather than having all threads go to
Hi all,
I have some doubts about arrow flight throughput. In this
article(https://www.dremio.com/understanding-apache-arrow-flight/), it said
"High efficiency. Flight is designed to work without any serialization or
deserialization of records, and with zero memory copies, achieving over 20 Gbp
13 matches
Mail list logo
