Hi Micah, I was wondering where the arrow project stands on this issue, as it looks like there are not many work arounds to using pyarrow.list_(pyarrow.struct()) as many other datatypes that would "fit the bill" of what a list of structs achieves raises a pyarrow.lib.ArrowNotImplementedError when calling table_to_blocks().
On Tue, Nov 8, 2022 at 12:53 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > Hi Matthew, > Could you give some more specifics about what language/component you are > using. In general, Arrow at a specification level doesn't deal with schema > evolution. Is this in regard to Datasets or a different component? > > Thanks, > Micah > > On Mon, Nov 7, 2022 at 5:06 PM Matthew Scanlon < > matthew.scan...@exosfinancial.com> wrote: > > > Good afternoon, I wanted to reach out and open a dialog about structs, > the > > evolution of them in schemas, and if support for such a feature is on the > > road map or a hard pass for the arrow team. > > > > Currently, it appears structs support removing a field, but will there be > > support for adding fields later on? Are there any recommended patterns > for > > supporting such a field. For example, if a field foo is a struct with > > sub_fields A, B and then later field C gets added, the old data can not > be > > loaded using the new schema. > > > > Thank you. > > > > Matthew Scanlon > > > > -- > > > > > > Broker-Dealer services offered through Exos Securities LLC, member of > > SIPC <http://www.sipc.org/> / FINRA <http://www.finra.org/> / > > BrokerCheck > > <https://brokercheck.finra.org/>/ 2022 Exos, inc. For important > > disclosures, click here > > <https://www.exosfinancial.com/general-disclosures>. > > > > > > > > > -- Broker-Dealer services offered through Exos Securities LLC, member of SIPC <http://www.sipc.org/> / FINRA <http://www.finra.org/> / BrokerCheck <https://brokercheck.finra.org/>/ 2022 Exos, inc. For important disclosures, click here <https://www.exosfinancial.com/general-disclosures>.
