Re: [C++][Parquet]: Stream API handling of optional fields

2019-11-16 Thread Micah Kornfield
Thanks Gawain. For reference, https://github.com/apache/arrow/pull/5849 is the PR. We might want to wait a day or two before merging to make sure no one has any objections to this approach approach. On Sat, Nov 16, 2019 at 6:51 AM Gawain Bolton wrote: > Thanks for your reply. > > If I understa

Re: [C++][Parquet]: Stream API handling of optional fields

2019-11-16 Thread Gawain Bolton
Thanks for your reply. If I understand correctly ARROW-7178 must be done so that Arrow has a version of std::optional which Parquet could then use. I think I will submit a PR for this shortly. Gawain On 15/11/2019 14:05, Francois Saint-Jacques wrote: I'm all for it. Created [1] it would als

Re: [C++][Parquet]: Stream API handling of optional fields

2019-11-15 Thread Francois Saint-Jacques
I'm all for it. Created [1] it would also enable an operator[] for arrays of primitive types [2]. [1] https://issues.apache.org/jira/browse/ARROW-7178 [2] https://issues.apache.org/jira/browse/ARROW-6276 On Fri, Nov 15, 2019 at 12:40 AM Micah Kornfield wrote: > > I think there are potentially ot

Re: [C++][Parquet]: Stream API handling of optional fields

2019-11-14 Thread Micah Kornfield
I think there are potentially other places in the Arrow code base that "optional" could be useful (e.g. a row-reader like class for Arrow Tables). It looks like there is at least 1 header only optional library [1] that is c++17 forward compatible. I think I would lean towards vendoring that or an

[C++][Parquet]: Stream API handling of optional fields

2019-11-14 Thread Gawain Bolton
Hello, I would like to add support for handling optional fields to the parquet::StreamReader and parquet::StreamWriter classes which I recently contributed (thank you!). Ideally I would do this by using std::optional like this:     parquet::StreamWriter writer{ parquet::ParquetFileWriter::Op