Re: [DISCUSS] Iceberg roadmap

Update: As the Dell EMC EcsFileIO has been merged into apache iceberg official repo, so I think it's okay to get this project from roadmap closed now: https://github.com/apache/iceberg/projects/22 Thanks. On Wed, Nov 10, 2021 at 10:22 AM Zhao Chun wrote: > Thanks Ryan. > We will keep a close e

Re: [DISCUSS] Iceberg roadmap

Thanks Ryan. We will keep a close eye on what is happening in the iceberg community and seek help when necessary. Thanks, Zhao Chun Ryan Blue 于2021年11月10日周三 上午8:54写道： > Thanks, Zhao. I think those are great ways to work together. Let us know > how we can help you make StarRocks successful with

Re: [DISCUSS] Iceberg roadmap

Thanks, Zhao. I think those are great ways to work together. Let us know how we can help you make StarRocks successful with Iceberg as its data format. We're always happy to help people understand how Iceberg works and improve our docs on how to use it. Ryan On Mon, Nov 8, 2021 at 8:17 PM Zhao Ch

Re: [DISCUSS] Iceberg roadmap

I feel that Ryan's response exemplifies the generosity of an Apache project creator, a quality that has touched and benefited us. We look forward to contributing further to the Apache project in the future. As for the need for an issue to track progress,I don't think so for now. At the moment the m

Re: [DISCUSS] Iceberg roadmap

I think it is great to see another processing engine adding support for Apache Iceberg, and I do look forward to collaborating with the StarRocks community in the future. I'm not entirely sure what that collaboration would look like just yet though. For most processing engines, it is people joinin

Re: [DISCUSS] Iceberg roadmap

Thanks to @OpenInx for mentioning StarRocks in the iceberg community. I'm from the StarRocks community. StarRocks is based on the Apache Doris project. It has been in development internally for almost two years and is currently used by hundreds of companies. It was just opened 2 months ago. Iceb

Re: [DISCUSS] Iceberg roadmap

+1 around concerns with the Elastic license. Also, more importantly, how important is integration with either of these tools to the Iceberg community and contributors? The Elastic license makes a bit more sense for elasticsearch, as it was an existing project for quite some time. I won’t reiterat

Re: [DISCUSS] Iceberg roadmap

+1, I have the same concern for the incompatible license. Jacques Nadeau 于2021年11月8日周一 上午11:48写道： > A few additional observations about StarRocks... > > - As far as I can tell, StarRocks has an ASF incompatible license (Elastic > License 2.0). > - It appears to be a hard fork of Apache Doris, a

Re: [DISCUSS] Iceberg roadmap

A few additional observations about StarRocks... - As far as I can tell, StarRocks has an ASF incompatible license (Elastic License 2.0). - It appears to be a hard fork of Apache Doris, a project still in the incubator (and looks like it probably is destructive to the Doris project) - The project

Re: [DISCUSS] Iceberg roadmap

Any thoughts for adding StarRocks integration to the roadmap ? I think the guys from StarRocks community can provide more background and inputs. On Thu, Nov 4, 2021 at 5:59 PM OpenInx wrote: > Update: > > StarRocks[1] is a next-gen sub-second MPP database for full analysis > scenarios, includin

Re: [DISCUSS] Iceberg roadmap

Update: StarRocks[1] is a next-gen sub-second MPP database for full analysis scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query. Their team is planning to integrate iceberg tables as StarRocks external tables in the next month [2], so that people could connect

Re: [DISCUSS] Iceberg roadmap

I closed the upgrade project and marked the FLIP-27 project priority 1. Thanks for all the work to get this done! On Sun, Oct 31, 2021 at 8:10 PM OpenInx wrote: > Update: > > I think the project [Flink: Upgrade to 1.13.2][1] in RoadMap can be > closed now, because all of the issues have been ad

Re: [DISCUSS] Iceberg roadmap

Update: I think the project [Flink: Upgrade to 1.13.2][1] in RoadMap can be closed now, because all of the issues have been addressed. [1]. https://github.com/apache/iceberg/projects/12 On Tue, Sep 21, 2021 at 6:17 PM Eduard Tudenhoefner wrote: > I created a Roadmap section in https://github.

Re: [DISCUSS] Iceberg roadmap

I created a Roadmap section in https://github.com/apache/iceberg/pull/3163 that links to the planning boards that Jack created. I figured it makes sense if we link available Design Docs directly on those Boards (as was already done), because then the De

Re: [DISCUSS] Iceberg roadmap

Thanks, Jack! Eduard, I think that's a good idea. We should have a roadmap page as well that links to the projects that Jack just created. On Mon, Sep 20, 2021 at 12:57 PM Jack Ye wrote: > It seems like we have reached some consensus around the projects listed > here. I have created correspondi

Re: [DISCUSS] Iceberg roadmap

It seems like we have reached some consensus around the projects listed here. I have created corresponding Github projects for each: https://github.com/apache/iceberg/projects Related design docs are also linked there. Best, Jack Ye On Sun, Sep 19, 2021 at 11:18 PM Eduard Tudenhoefner wrote: >

Re: [DISCUSS] Iceberg roadmap

Would it make sense to have a section on the website where we collect all the links to the design docs/specs as that would be easier to find than searching for things on the ML? I was thinking about something like for each component: * link to the ML discussion * link to the actual Spec/Design Doc

Re: [DISCUSS] Iceberg roadmap

Thanks Steven & Kyle. Yes, the flip-27 source and flink 1.13.2 are orthogonal because the flink's flip-27 API was successfully introduced in flink 1.12 release ( https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface). The WIP flip-27 iceberg source proposed from

Re: [DISCUSS] Iceberg roadmap

This list looks overall pretty good to me. +1 For Flink 1.13 upgrade, I suggest we consider starting another thread for it. There are some open PRs, but they have outstanding questions. Specifically, dropping support for Flink 1.12 or not. I think we can upgrade without dropping support for Fli

Re: [DISCUSS] Iceberg roadmap

That sounds great, thanks for taking that on Jack! On Wed, Sep 15, 2021 at 3:51 PM Jack Ye wrote: > For external Trino and PrestoDB tasks, I am thinking about creating one > Github project for Trino and another one for PrestoDB to manage all tasks > under them, adding links of issues and PRs in

Re: [DISCUSS] Iceberg roadmap

For external Trino and PrestoDB tasks, I am thinking about creating one Github project for Trino and another one for PrestoDB to manage all tasks under them, adding links of issues and PRs in the other communities to track progress. This is mostly to improve visibility so that people who are intere

Re: [DISCUSS] Iceberg roadmap

Gidon, I think that the v3 part of encryption is actually documenting how it works and adding it to the spec. Right now we have hooks for building some encryption around it, but almost no requirements in the spec for how to use it across implementations. This is fine while we're working on defining

Re: [DISCUSS] Iceberg roadmap

The discussed roadmap makes sense to me. I think it is important to agree on what we should do first as the review pool is limited. There are more and more large items that are half done or half discussed. I think we better focus on finishing them quickly and then move to something else as oppos

Re: [DISCUSS] Iceberg roadmap

I'd like to also propose adding the following in the external section: 1. the PrestoDB equivalent for each item listed for Trino. I am not sure what's the best way to track them, but I feel it's better to list and track them separately. I have talked with related people currently maintaining the Pr

Re: [DISCUSS] Iceberg roadmap

Hi Ryan, I just wonder if the encryption should be a Spec v3 category. We have the key_metadata fields in both data_file and manifest_file structs, which might be sufficient for a reasonable basic encryption support. But I certainly agree this is an L-sized project. Cheers, Gidon On Sat, Sep 11

Re: [DISCUSS] Iceberg roadmap

Ryan, FLIP-27 Flink source is not part of the Flink 1.13.2 upgrade, although Flink 1.13.2 is preferred for some bug fix. On Sun, Sep 12, 2021 at 9:04 AM Ryan Blue wrote: > Szehon, I'm not sure that I'd consider either of those for the roadmap. > While they are important to work on, they seem mor

Re: [DISCUSS] Iceberg roadmap

Szehon, I'm not sure that I'd consider either of those for the roadmap. While they are important to work on, they seem more like individual PRs than high-level projects that might span PRs. This is definitely a good thing to consider, though. Should we go more granular than the list that we've comp

Re: [DISCUSS] Iceberg roadmap

Steven, I thought that this was part of upgrading Flink to 1.13.2. Is that correct? On Fri, Sep 10, 2021 at 4:40 PM Steven Wu wrote: > I would like to add a item > > Priority 2: > Flink: FLIP-27 based Iceberg source [large] > > On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue wrote: > >> Hi everyone,

Re: [DISCUSS] Iceberg roadmap

Hi I also missed the last sync, and wanted to add two things if possible. Thanks, Szehon Priority 2: - Core: Predicate pushdown for remaining Metadata tables [medium] - Core/Spark: Support serializable isolation for ReplacePartitions / Insert Overwrite [medium] On Fri, Sep 10, 2021 a

Re: [DISCUSS] Iceberg roadmap

I would like to add a item Priority 2: Flink: FLIP-27 based Iceberg source [large] On Fri, Sep 10, 2021 at 2:38 PM Ryan Blue wrote: > Hi everyone, > > At the last sync meeting, we brought up publishing a community roadmap and > brainstormed the many features and initiatives that the community i

[DISCUSS] Iceberg roadmap

Hi everyone, At the last sync meeting, we brought up publishing a community roadmap and brainstormed the many features and initiatives that the community is working on. In this thread, I want to make sure that we have a good list of what people are thinking about and I think we should try to categ