Re: Is it possible to add computed columns to a pyarrow dataset

2023-10-25 Thread Chang She
Do you already have a storage layer to persist these views or do you only need ephemeral views? Sounds interesting curious to find out more about your use case On Wed, Oct 25, 2023 at 2:00 PM Lee, David (PAG) wrote: > Here's my ideal use case scenario.. > > Create multiple datasets mapped to dif

Is it possible to add computed columns to a pyarrow dataset

2023-10-25 Thread Lee, David (PAG)
Here's my ideal use case scenario.. Create multiple datasets mapped to different file directories. Create more datasets by logically generating additional computed columns using expressions Create joined dataset by joining datasets Finally run a Scanner on the joined dataset to start materializa

Re: Interacting with Avro in Golang

2023-10-25 Thread Matt Topol via user
Sorry for the VERY late reply here, but there is an ongoing PR being worked on for Avro support in the Go Arrow libs [1]. Feel free to watch / contribute / test out that branch while we iron out the specifics and get it ready. --Matt [1]: https://github.com/apache/arrow/pull/37115 On Sun, Sep 3,

Re: [Java][Format] Support for Run End Encoded Vectors

2023-10-25 Thread Elliott Bradshaw
Over the wire helps for sure, but keeping memory pressure down is the real goal. Let me know if anyone decides to take this. Elliott Bradshaw *Co-Founder | Principal* *Tectonix, LLC* m: 443-285-9224 a: Columbia, MD w: www.tectonix.com e: elli...@tectonix.com CONFIDENTIALITY NOTICE: The contents