Re: [EXT] Re: Struct evolution

2022-12-14 Thread Matthew Scanlon
Hello, I just wanted to follow up on our previous conversation and gain a bit more insight into the behavior of pyarrow tables reading from/writing to dataframes. I have noticed some interesting behavior related to the issue mentioned above; specifically that this does not seem to be an issue when

Re: [EXT] Re: Struct evolution

2022-11-28 Thread Matthew Scanlon
Hi Micah, I was wondering where the arrow project stands on this issue, as it looks like there are not many work arounds to using pyarrow.list_(pyarrow.struct()) as many other datatypes that would "fit the bill" of what a list of structs achieves raises a pyarrow.lib.ArrowNotImplementedError when c

Re: [EXT] Re: Struct evolution

2022-11-14 Thread Matthew Scanlon
Hi Micah. Sorry for the late reply as I have been on holiday. I am referring to datasets. And this was specifically noticed in python though I would imagine the issue can be abstracted to other languages as well On Tue, Nov 8, 2022 at 12:53 PM Micah Kornfield wrote: > Hi Matthew, > Could you gi