Hey Tim, Maybe you can shed some light on this for me. Again sorry if this is well know but I just found out about perspective and I have been playing around with it.
Is the thought that the output of to_arrow() should not be used in non perspective context? For my use case we are thinking of using perspective in Jupyter notebooks to build up complex representation of tabular data involving pivoting and filtering. And we would take that output of to_arrow and feed into some of our own rendering engines. It did seem like the arrow output changed when I did pivoting, but I could not figure out how or why, haha. It seems that the schema changes to arrow is a custom solution for just Perspective and it might be prudent to wait for Arrow 4 that will have a standard way of representing this information. -Michael From: Tim Paine <t.paine...@gmail.com> Date: Friday, March 19, 2021 at 9:53 AM To: dev@arrow.apache.org <dev@arrow.apache.org> Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow Perspective uses arrow across the wire but internally uses it's own formats. Tim Paine tim.paine.nyc 908-721-1185 > On Mar 19, 2021, at 09:46, Michael Lavina <michael.lav...@factset.com> wrote: > > Hey Benjamin, > > That sounds really awesome. Thank you. > > Sorry if this was already a well known thing as I am fairly new to the Arrow > ecosystem. Is there a way to track a roadmap for Arrow 4 and be involved in > that? Is there anywhere I can read more just general information on that? > > -Michael > > From: Benjamin Kietzman <bengil...@gmail.com> > Date: Friday, March 19, 2021 at 9:14 AM > To: dev <dev@arrow.apache.org> > Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow > Hi Michael, > > We are targeting grouped aggregation for 4.0 as part of a general query > engine buildout. We also intend to bring DataFrame functionality into core > Arrow (which would probably include an analog of pandas' pivot_table), but > the query engine work is a prerequisite. > > Ben Kietzman > >> On Fri, Mar 19, 2021, 08:19 Michael Lavina <michael.lav...@factset.com> >> wrote: >> >> Hey Team, >> >> Sorry if this is answered already somewhere I tried searching emails and >> issues but couldn’t find anything. I am wondering if there is a standard >> way to encode row or column pivots in Arrow? >> >> I know Pandas does it already some way >> https://urldefense.com/v3/__https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$<https://urldefense.com/v3/__https:/pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$<https://urldefense.com/v3/__https:/pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$%3chttps:/urldefense.com/v3/__https:/pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$>> >> and there are libraries using Arrow like Perspective that may have their >> internal solution for representation of pivots >> https://urldefense.com/v3/__https://perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$<https://urldefense.com/v3/__https:/perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$<https://urldefense.com/v3/__https:/perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$%3chttps:/urldefense.com/v3/__https:/perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$>> >> >> I am wondering if there is already a discussion or already a best practice >> or standard for encoding this information. Or alternatively is this not >> something that should be at all associated with Arrow. >> >> -Michael >> >> P.S. If anyone on the Perspective team or anyone who might know is on this >> thread I would be interested in understanding more how Perspective, >> specifically, encodes pivot information in Arrow. >> >>
