Re: [DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Wes McKinney
o track a roadmap for Arrow 4 and be > > involved in that? Is there anywhere I can read more just general > > information on that? > > > > -Michael > > > > From: Benjamin Kietzman > > Date: Friday, March 19, 2021 at 9:14 AM > > To: dev > > Subje

Re: [DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Michael Lavina
@arrow.apache.org Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow Perspective uses arrow across the wire but internally uses it's own formats. Tim Paine tim.paine.nyc 908-721-1185 > On Mar 19, 2021, at 09:46, Michael Lavina wrote: > > Hey Benjamin, &

Re: [DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Tim Paine
9:14 AM > To: dev > Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow > Hi Michael, > > We are targeting grouped aggregation for 4.0 as part of a general query > engine buildout. We also intend to bring DataFrame functionality into core > Arrow (which

Re: [DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Michael Lavina
From: Benjamin Kietzman Date: Friday, March 19, 2021 at 9:14 AM To: dev Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow Hi Michael, We are targeting grouped aggregation for 4.0 as part of a general query engine buildout. We also intend to bring DataFrame functionality

Re: [DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Benjamin Kietzman
Hi Michael, We are targeting grouped aggregation for 4.0 as part of a general query engine buildout. We also intend to bring DataFrame functionality into core Arrow (which would probably include an analog of pandas' pivot_table), but the query engine work is a prerequisite. Ben Kietzman On Fri,

[DISCUSS] How to encode table_pivot information state in Arrow

2021-03-19 Thread Michael Lavina
Hey Team, Sorry if this is answered already somewhere I tried searching emails and issues but couldn’t find anything. I am wondering if there is a standard way to encode row or column pivots in Arrow? I know Pandas does it already some way https://pandas.pydata.org/pandas-docs/stable/reference