On Thu, Feb 25, 2016 at 12:05 PM, Henry Robinson wrote:
> The way I'm thinking about is that someone upstream makes a Kudu-specific
> request, but as part of that request provides a descriptor of a shared
> ring-buffer. Reading Arrow batches from and writing to that buffer is part
> of a simple st
reed,
> >> >> > > >
> >> >> > > > I thought the whole purpose was to share the memory space
> (using
> >> >> > possibly
> >> >> > > > unsafe operations like ByteBuffers) so that it could be
> directly
> >> >> shared
> >> >
gt; > > datastore that one application can expose and share with others
>> (e.g.
>> >> > In
>> >> > > > memory structure is constructed from a series of parquet files,
>> >> > somehow,
>> >> > > > then Spark pulls it i
datastore that one application can expose and share with others
> (e.g.
> >> > In
> >> > > > memory structure is constructed from a series of parquet files,
> >> > somehow,
> >> > > > then Spark pulls it in, does some computations, exposes a d
mputations, exposes a data set,
>> > > > etc...).
>> > > >
>> > > > If you are leaving the allocation of the memory to the applications
>> and
>> > > > underneath the memory is being allocated using direct bytebuffers, I
>> > > ca
tally hard- especially if the
> > > > applications themselves are worried about exposing their own memory
> > > spaces.
> > > >
> > > >
> > > > On Wed, Feb 24, 2016 at 2:17 PM, Andrew Brust <
> > > > andrew.br...@bluebadgein
4, 2016 at 2:17 PM, Andrew Brust <
> > > andrew.br...@bluebadgeinsights.com> wrote:
> > >
> > > > Hmm...that's not exactly how Jaques described things to me when he
> > > briefed
> > > > me on Arrow ahead of the announcement.
> > >
..@bluebadgeinsights.com> wrote:
> >
> > > Hmm...that's not exactly how Jaques described things to me when he
> > briefed
> > > me on Arrow ahead of the announcement.
> > >
> > > -Original Message-
> > > From: Zhe Zhang [mailto:z...@apa
on Arrow ahead of the announcement.
> >
> > -Original Message-
> > From: Zhe Zhang [mailto:z...@apache.org]
> > Sent: Wednesday, February 24, 2016 2:08 PM
> > To: dev@arrow.apache.org
> > Subject: Re: Question about mutability
> >
> > I don&#x
in the case of shared in-memory arrow caching
> > layers as those get created.
> >
> > Back to the original question about mutability:
> >
> > These structures are designed to be write once, read many. So once they
> are
> > constructed, they are immutable.
>
ch
taking turns working with the data, in a cumulative fashion.
-Original Message-
From: Michael D. Coon [mailto:mdco...@yahoo.com.INVALID]
Sent: Wednesday, February 24, 2016 2:21 PM
To: dev@arrow.apache.org
Subject: Re: Question about mutability
I had the same understanding as Corey
pectation is that in most cases the consumer of the structures will
> provide memory for purposes of population by the provider. That being said,
> we'll need to flip this in the case of shared in-memory arrow caching
> layers as those get created.
>
> Back to the original question
o: dev@arrow.apache.org
> Subject: Re: Question about mutability
>
> I don't think one application/process's memory space will be made
> available to other applications/processes. It's fundamentally hard for
> processes to share their address spaces.
>
> IIUC,
ition. My
expectation is that in most cases the consumer of the structures will
provide memory for purposes of population by the provider. That being said,
we'll need to flip this in the case of shared in-memory arrow caching
layers as those get created.
Back to the original question about
rote:
>
>
> Hmm...that's not exactly how Jaques described things to me when he
> briefed me on Arrow ahead of the announcement.
>
> -Original Message-
> From: Zhe Zhang [mailto:z...@apache.org]
> Sent: Wednesday, February 24, 2016 2:08 PM
> To: dev@arrow.apach
rg
Subject: Re: Question about mutability
I don't think one application/process's memory space will be made available to
other applications/processes. It's fundamentally hard for processes to share
their address spaces.
IIUC, with Arrow, when application A shares data with applicat
Hmm...that's not exactly how Jaques described things to me when he briefed me
on Arrow ahead of the announcement.
-Original Message-
From: Zhe Zhang [mailto:z...@apache.org]
Sent: Wednesday, February 24, 2016 2:08 PM
To: dev@arrow.apache.org
Subject: Re: Question about mutabili
I don't think one application/process's memory space will be made available
to other applications/processes. It's fundamentally hard for processes to
share their address spaces.
IIUC, with Arrow, when application A shares data with application B, the
data is still duplicated in the memory spaces o
Forgive me if this question seems ill-informed. I just started looking at
Arrow yesterday. I looked around the github a tad.
Are you expecting the memory space held by one application to be mutable by
that application and made available to all applications trying to read the
memory space?
19 matches
Mail list logo
