Re: [DISCUSS] Iceberg roadmap

2021-09-10 Thread Szehon Ho
Hi I also missed the last sync, and wanted to add two things if possible. Thanks, Szehon Priority 2: - Core: Predicate pushdown for remaining Metadata tables [medium] - Core/Spark: Support serializable isolation for ReplacePartitions / Insert Overwrite [medium] On Fri, Sep 10, 2021 a

Re: [DISCUSS] Iceberg roadmap

2021-09-10 Thread Steven Wu
I would like to add a item Priority 2: Flink: FLIP-27 based Iceberg source [large] On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue wrote: > Hi everyone, > > At the last sync meeting, we brought up publishing a community roadmap and > brainstormed the many features and initiatives that the community i

[DISCUSS] September board report

2021-09-10 Thread Ryan Blue
Hi everyone, Here’s what I’ve compiled for our September board report. Please reply if you’d like to add or clarify anything! Ryan Description: Apache Iceberg is a table format for huge analytic datasets that is designed for high performance and ease of use. Issues: There are no issues requirin

[DISCUSS] Iceberg roadmap

2021-09-10 Thread Ryan Blue
Hi everyone, At the last sync meeting, we brought up publishing a community roadmap and brainstormed the many features and initiatives that the community is working on. In this thread, I want to make sure that we have a good list of what people are thinking about and I think we should try to categ

Re: Iceberg community sync notes for 1 September 2021

2021-09-10 Thread Ryan Blue
Thanks for adding these items, @OpenInx! I'll add them to the roadmap discussion. One thing I want to note is that I don't think that the roadmap is going to be tied to releases. Since the start, we've not delayed Iceberg releases to get specific features in. Instead, we've preferred to get releas

Re: A new project focused on serialized algebra

2021-09-10 Thread Jacques Nadeau
There are also some good conversations happening in on the Github discussion forum [1]. We're trying to drive consensus around the type system to start [2]. Would love the Iceberg community members to weigh in as the contributors are fairly Arrow heavy atm. Thanks, Jacques [1] https://github.co

Re: A new project focused on serialized algebra

2021-09-10 Thread Ryan Blue
Nevermind, I see there's a Substrait Slack community. Here's the invite link for anyone else that's interested: https://join.slack.com/t/substrait/shared_invite/zt-vivbux2c-~B1jEWcR0wYhq5k4LHuoLQ On Fri, Sep 10, 2021 at 9:16 AM Ryan Blue wrote: > Thanks, Jacques! I think it's a great idea to hav

Re: A new project focused on serialized algebra

2021-09-10 Thread Ryan Blue
Thanks, Jacques! I think it's a great idea to have this as an external project so that it doesn't get tied to a particular set of goals for an existing project. Where is a good place to discuss this? Should we create a #substrait room on Iceberg Slack? ASF Slack? On this thread? Ryan On Wed, Sep