We do this [1] in the R package to use RecordBatchReaders as input (it
powers the DuckDB integration and a few other things we need to evaluate R
things that don't have a corresponding Acero node).

[1]
https://github.com/apache/arrow/blob/master/r/src/compute-exec.cpp#L396-L406
to

On Wed, Sep 7, 2022 at 1:31 PM Wes McKinney <wesmck...@gmail.com> wrote:

> This seems like something where there should be ready-to-go code in
> the Arrow codebase to feed any RecordBatchReader into Acero
>
> On Thu, Aug 18, 2022 at 12:15 PM Li Jin <ice.xell...@gmail.com> wrote:
> >
> > Thanks all. I will try this out.
> >
> > On Thu, Aug 18, 2022 at 9:06 AM Rok Mihevc <rok.mih...@gmail.com> wrote:
> >
> > > +1 for adding this either a utility function or cookbook recipe [1].
> > >
> > > [1] https://github.com/apache/arrow-cookbook
> > >
> > > On Thu, Aug 18, 2022 at 2:34 PM Yaron Gvili <rt...@hotmail.com> wrote:
> > >
> > > > I have code in source_node.cc in a local branch adding factories for
> > > other
> > > > sources in SourceNode (e.g., streams of RecordBatch, ExecBatch, or
> > > > ArrayVector) which I could make a PR for, if there is interest.
> > > >
> > > >
> > > > Yaron.
> > > > ________________________________
> > > > From: David Li <lidav...@apache.org>
> > > > Sent: Wednesday, August 17, 2022 4:25 PM
> > > > To: dev@arrow.apache.org <dev@arrow.apache.org>
> > > > Subject: Re: [C++] Read Flight data source into Acero
> > > >
> > > > Convert the reader to a RecordBatchReader [1], turn it into an
> > > > AsyncGenerator [2], then wrap it in a SourceNode (though I think
> > > SourceNode
> > > > should just expose a helper for wrapping a RecordBatchReader like it
> > > > already does for Tables).
> > > >
> > > > [1]:
> > > >
> > >
> https://github.com/apache/arrow/blob/8474ee5a3ed725d4bb56c75fc1b13a53cba1fd1f/cpp/src/arrow/flight/types.h#L561
> > > > [2]:
> > > >
> > >
> https://github.com/apache/arrow/blob/ee2e9448c8565820ba38a2df9e44ab6055e5df1d/cpp/src/arrow/compute/exec/exec_plan.h#L529-L535
> > > >
> > > > On Wed, Aug 17, 2022, at 15:32, Aldrin wrote:
> > > > > I don't have any pointers, but just wanted to mention that I am
> going
> > > to
> > > > > try and figure this out quite a bit in the next week. I can try to
> > > create
> > > > > some relevant cookbook recipes as I plod along.
> > > > >
> > > > > Aldrin Montana
> > > > > Computer Science PhD Student
> > > > > UC Santa Cruz
> > > > >
> > > > >
> > > > > On Wed, Aug 17, 2022 at 9:15 AM Li Jin <ice.xell...@gmail.com>
> wrote:
> > > > >
> > > > >> Correction: I have a flight::FlightStreamReader (not
> > > > Flight::StreamReader)
> > > > >>
> > > > >> On Wed, Aug 17, 2022 at 12:12 PM Li Jin <ice.xell...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > I have a Flight data source (effectively a
> Flight::StreamReader) and
> > > > I'd
> > > > >> > like to create an Acero source node from it. I wonder if
> something
> > > > >> already
> > > > >> > exists to do that or if not, perhaps some pointers for me  to
> take a
> > > > look
> > > > >> > at?
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Li
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
>

Reply via email to