Re: Welcome Huaxin Gao as a committer!

;>>> > >>>  >>>>>>>>> > >>> >>>>>>>>> > >>> Congratulations Huaxin! >>>>>>>>> > >>> >>>>>>>>> > >>> Op do 6 feb 2025 om 12:21 schreef Russell Spitzer < >>>>>>>>> > >>> russell.spit...@gmail.com>: >>>>>>>>> > >>> >>>>>>>>> > >>>> Congratulations! >>>>>>>>> > >>>> >>>>>>>>> > >>>> On Thu, Feb 6, 2025 at 11:35 AM Péter Váry < >>>>>>>>> peter.vary.apa...@gmail.com> >>>>>>>>> > >>>> wrote: >>>>>>>>> > >>>> >>>>>>>>> > >>>>> Congratulations! >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> Matt Topol ezt írta (időpont: >>>>>>>>> 2025. febr. >>>>>>>>> > >>>>> 6., Cs, 10:40): >>>>>>>>> > >>>>> >>>>>>>>> > >>>>>> Congrats! Welcome! >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> On Thu, Feb 6, 2025, 10:19 AM Raúl Cumplido < >>>>>>>>> rau...@apache.org> >>>>>>>>> > >>>>>> wrote: >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>> Congrats Huaxin! >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> El jue, 6 feb 2025 a las 10:16, Gang Wu (< >>>>>>>>> ust...@gmail.com>) >>>>>>>>> > >>>>>>> escribió: >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>> Congrats Huaxin! >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> Best, >>>>>>>>> > >>>>>>>> Gang >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> On Thu, Feb 6, 2025 at 5:10 PM Szehon Ho < >>>>>>>>> szehon.apa...@gmail.com> >>>>>>>>> > >>>>>>>> wrote: >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>> Hi everyone, >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> The Project Management Committee (PMC) for Apache >>>>>>>>> Iceberg has >>>>>>>>> > >>>>>>>>> invited Huaxin Gao to become a committer, and I am >>>>>>>>> happy to >>>>>>>>> > >>>>>>>>> announce that she has accepted. Huaxin has done a lot >>>>>>>>> > >>>>>>>>> of impressive work in areas such as Iceberg-Spark >>>>>>>>> integration and recently >>>>>>>>> > >>>>>>>>> Iceberg-Comet integrations. Thanks Huaxin for all >>>>>>>>> your hard work! >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> Please join us in welcoming her! >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> Thanks, >>>>>>>>> > >>>>>>>>> Szehon >>>>>>>>> > >>>>>>>>> On behalf of the Iceberg PMC >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Himadri Pal >>>>>>>> >>>>>>> >>> > -- John Zhuge

Re: [DISCUSS/VOTE] Add in ChangeLog Reserved Field IDs to Spec and Decrement Row Lineage Reserved IDs

+1 (non-binding) John Zhuge On Sun, Jan 26, 2025 at 2:59 PM Aihua Xu wrote: > +1 (non-binding). > > Thanks for fixing it. > > On Sun, Jan 26, 2025 at 11:30 AM Anton Okolnychyi > wrote: > >> +1 good catch >> >> нд, 26 січ. 2025 р. о 08:33 Steve

Re: [DISCUSS] Add a implementation status page for iceberg

t;>>> spec >>>>>>>>> though. >>>>>>>>> >>>>>>>>> Updated the doc, please take a look. >>>>>>>>> >>>>>>>>> > My only comment is probably to use versions instead of check >>>>>>>>> marks, but all good :) >>>>>>>>> >>>>>>>>> In current approach we will write the version of each library in >>>>>>>>> the beginning of the page, which seems easier to maintain than per >>>>>>>>> version >>>>>>>>> per feature. What do you think? >>>>>>>>> >>>>>>>>> On Sat, Nov 9, 2024 at 5:12 PM Jean-Baptiste Onofré < >>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I like the idea. My only comment is probably to use versions >>>>>>>>>> instead >>>>>>>>>> of check marks, but all good :) >>>>>>>>>> >>>>>>>>>> Thanks ! >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> JB >>>>>>>>>> >>>>>>>>>> On Fri, Nov 8, 2024 at 3:33 PM Russell Spitzer >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Sounds like a great idea to me >>>>>>>>>> > >>>>>>>>>> > On Fri, Nov 8, 2024 at 7:58 AM Renjie Liu < >>>>>>>>>> liurenjie2...@gmail.com> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Hi: >>>>>>>>>> >> >>>>>>>>>> >> As iceberg evolved to a multi-lang project, I would like to >>>>>>>>>> propose to maintain a status page for iceberg. For more details, >>>>>>>>>> please >>>>>>>>>> refer to this doc. Welcome to join the discussion and comment on it! >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>> -- John Zhuge

Re: [DISCUSS] Standardizing Error Handling in the Iceberg Spark Module

+1 Thanks John Zhuge On Thu, Dec 19, 2024 at 9:47 AM Yufei Gu wrote: > +1 on the direction. It's great that Spark has standardized the error code > so that Iceberg didn't have to rely on error messages. > > Yufei > > > On Thu, Dec 19, 2024 at 8:47 AM rdb...@gm

Re: [DISCUSS] Apache Iceberg Summit 2025 - Selection Committee

tion committee that will be >> > >>>> responsible for choosing talks and guiding the process. >> > >>>> Once we have a selection committee, I will complete the concrete >> > >>>> proposal for the ASF and the Iceberg PMC to request the ability to >> use >> > >>>> the name Iceberg/Apache Iceberg. >> > >>>> >> > >>>> If you'd like to help and be part of the selection committee, >> please >> > >>>> volunteer in a reply to this thread. Since we likely can't include >> > >>>> everyone that volunteers, I propose that the PMC should choose the >> > >>>> final committee from the set of people that volunteer. >> > >>>> >> > >>>> We'll leave this open up to Dec 10th to give people time (as >> > >>>> Thanksgiving is this week). >> > >>>> >> > >>>> Thanks ! >> > >>>> Regards >> > >>>> JB >> > >>>> >> > >>> >> > >> >> > >> -- >> > >> Regards, >> > >> Himadri Pal >> > >> >> > > >> > >> > -- John Zhuge

Re: [DISCUSS] Add a implementation status page for iceberg

ed to a multi-lang project, I would like to propose to >>>> maintain a status page for iceberg. For more details, please refer to this >>>> doc >>>> <https://docs.google.com/document/d/1sRsTatGQJJNiBiQZNUW4VwQDCV1e75BHM6cSPla4vBU/edit?usp=sharing>. >>>> Welcome to join the discussion and comment on it! >>>> >>>> >>>> -- John Zhuge

Re: [VOTE] Deletion Vectors in V3

t;>>>>>> >>>>>>>>> Regards >>>>>>>>> JB >>>>>>>>> >>>>>>>>> On Tue, Oct 29, 2024 at 10:45 PM Anton Okolnychyi < >>>>>>>>> aokolnyc...@gmail.

Re: [VOTE] Endpoint for refreshing vended credentials

+1 (non-binding) John Zhuge On Tue, Oct 22, 2024 at 9:45 AM Jack Ye wrote: > +1 (binding) > > Best, > Jack Ye > > On Tue, Oct 22, 2024 at 9:32 AM Dmitri Bourlatchkov > wrote: > >> Thanks for the reply Eduard! >> >> I think it is fine to defer fine-tu

Re: [DISCUSS] Remove iceberg-pig module ?

ly we should drop it >>>>> >>>>> On Thu, Oct 17, 2024 at 8:07 AM Jean-Baptiste Onofré >>>>> wrote: >>>>> >>>>>> Hi folks, >>>>>> >>>>>> Even if it seems the project is pretty close to 0.18 release, Apache >>>>>> Pig is a "dormant" project. >>>>>> >>>>>> I would like to discuss here if it would not make sense to remove the >>>>>> iceberg-pig module. >>>>>> >>>>>> Thoughts ? >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>> -- John Zhuge

Re: [DISCUSS] Iceberg Summit 2025 ?

+1 for a hybrid event John Zhuge On Mon, Sep 30, 2024 at 10:12 AM Sung Yun wrote: > Hi JB, thank you for starting this thread! > > I’m already very excited at the prospect of connecting with other members > of the community. I think it would be nice to organize one next year

Re: [ANNOUNCE] Apache Iceberg release 1.6.1

Thanks! John Zhuge On Wed, Aug 28, 2024 at 9:37 AM Steven Wu wrote: > Thanks Carl for driving this release! > > On Wed, Aug 28, 2024 at 8:34 AM Carl Steinbach wrote: > >> I'm pleased to announce the release of Apache Iceberg 1.6.1! >> >> Apache Icebe

Re: [VOTE] Spec changes in preparation for v3

ansforms >>>> * Reset heading levels that were set to de-clutter the TOC in previous >>>> site frameworks >>>> >>>> This will be open for at least 72 hours. >>>> >>>> [ ] +1 >>>> [ ] -0 >>>> [ ] -1 do not make these changes because . . . >>>> >>>> -- >>>> Ryan Blue >>>> >>> -- John Zhuge

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

uals join us. >>>>> >>>>> So without further ado, please welcome Péter Váry, Amogh Jahagirdar >>>>> and Eduard Tudenhoefner to the Apache Iceberg PMC. >>>>> >>>>> As usual I am excited about the future of this community and thankful >>>>> for the hard work and stewardship of its members. >>>>> >>>>> Thank you for your time, >>>>> Russell Spitzer >>>>> >>>> >>> >> >> -- >> Ryan Blue >> Databricks >> > -- John Zhuge

Re: [VOTE] Clarify "File System Tables" in the table spec

> +1 (non-binding) >>>>>>>>>> >>>>>>>>>> On Wed, Jul 31, 2024 at 5:12 PM Ryan Blue >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> As promised in the discussion thread, I've opened a PR to >>>>>>>>>>> clarify the "File System Tables" section and mark it deprecated >>>>>>>>>>> since there >>>>>>>>>>> appears to be consensus for at least warning people that this is >>>>>>>>>>> unsafe in >>>>>>>>>>> most cases and discouraged. >>>>>>>>>>> >>>>>>>>>>> The PR is here: https://github.com/apache/iceberg/pull/10833 >>>>>>>>>>> >>>>>>>>>>> Please vote on this spec change. This will be open for at least >>>>>>>>>>> 72 hours: >>>>>>>>>>> [] +1 >>>>>>>>>>> [] +0 >>>>>>>>>>> [] -1, do not deprecate File System Tables because . . . >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Ryan Blue >>>>>>>>>>> >>>>>>>>>> -- >>>> Robert Stupp >>>> @snazy >>>> >>>> >>> >>> -- >>> Ryan Blue >>> Databricks >>> >> -- John Zhuge

Re: Building with JDK 21

gt;>>>> > For Hive 2.x and 3.x, we have code in the Iceberg repo that is not >>>>>> being developed. Hive 2 is fairly easy since it is EOL. While Hive 3 is >>>>>> still used, I don't think it makes sense to keep releasing versions of it >>>>>> if it requires Java 8, which has not been publicly maintained for 5 >>>>>> years. >>>>>> We need to upgrade and that is at odds with keeping support for Hive 3. >>>>>> As >>>>>> Fokko and I both pointed out, people can still use older releases. >>>>>> > >>>>>> > For the question of how to maintain support for Hive 4, I think >>>>>> it's worth having a separate discussion (probably not on the thread about >>>>>> JDK versions) about where to maintain it. I think that it is best to >>>>>> maintain integration in engines and not in the Iceberg project; there are >>>>>> few implementations here and I think that it is a hard problem for >>>>>> Iceberg >>>>>> to maintain support for multiple versions (as you can see with support >>>>>> for >>>>>> so many different Flink, Hive, and Spark versions). >>>>>> > >>>>>> > Ryan >>>>>> > >>>>>> > On Thu, Jul 18, 2024 at 7:25 AM Denys Kuzmenko < >>>>>> dkuzme...@apache.org> wrote: >>>>>> >> >>>>>> >> In the following 1-2 months we plan to release HIVE-4.0.1 which >>>>>> includes bug fixes and then focus on HIVE-4.1.0 release with jdk17. >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Ryan Blue >>>>>> > Databricks >>>>>> >>>>> >> >> -- >> Ryan Blue >> Databricks >> > -- John Zhuge

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

Appreciate the thoughtful comments! On Thu, Jul 18, 2024 at 10:29 AM Jack Ye wrote: > Thank you for bringing this up Ryan. I have been also in the camp of > saying HadoopCatalog is not recommended, but after thinking about this more > deeply last night, I now have mixed feelings about this to

Re: [VOTE] spec: remove the JSON spec for content file and file scan task sections

+1 (non binding) John Zhuge On Wed, Jul 10, 2024 at 6:57 PM Xuanwo wrote: > +1 non-binding. > > The ieceberg-rust project doesn't refer to this too. > > On Thu, Jul 11, 2024, at 09:54, Renjie Liu wrote: > > +1 (non binding) > > On Thu, Jul 11, 2024 at 7:

Re: Materialized Views: Next Steps

gt;>>>>>> in at >>>>>>>>> this point. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://lists.apache.org/thread/rotmqzmwk5jrcsyxhzjhrvcjs5v3yjcc >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Walaa. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, May 8, 2024 at 2:31 AM Jan Kaul >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The original google doc >>>>>>>>>> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing> >>>>>>>>>> discussed multiple aspects of the Materialized View spec. One was the >>>>>>>>>> storage model while others were related to the metadata. After we >>>>>>>>>> (Micah, >>>>>>>>>> Szehon, you, me) reached consensus in the google doc, Jack raised his >>>>>>>>>> concern about the storage model and the long discussion about the >>>>>>>>>> storage >>>>>>>>>> model started. Now we truly reached consensus about the storage >>>>>>>>>> model, >>>>>>>>>> which is now also reflected in the google doc. All other aspects >>>>>>>>>> from the >>>>>>>>>> google doc about the metadata weren't questioned and still represent >>>>>>>>>> the >>>>>>>>>> consensus. >>>>>>>>>> >>>>>>>>>> I would like to *avoid repeating the discussions* in your PR >>>>>>>>>> that we already had in the google doc. Especially since we reached >>>>>>>>>> consensus which took a considerable amount of time. >>>>>>>>>> >>>>>>>>>> Thanks, Jan >>>>>>>>>> On 08.05.24 10:21, Walaa Eldin Moustafa wrote: >>>>>>>>>> >>>>>>>>>> Thanks Jan. I think we moved on to more alignment steps beyond >>>>>>>>>> that doc a while ago. After that doc, we have discussed the topic >>>>>>>>>> further >>>>>>>>>> in 2 dev list threads and one more doc >>>>>>>>>> <https://docs.google.com/document/d/1zg0wQ5bVKTckf7-K_cdwF4mlRi6sixLcyEh6jErpGYY/edit?pli=1> >>>>>>>>>> (with strictly two options for the storage model to consider). >>>>>>>>>> Moreover, >>>>>>>>>> the original doc grew to 14 pages long with one section comparing 5 >>>>>>>>>> design >>>>>>>>>> alternatives, which made things harder to reach consensus. The lack >>>>>>>>>> of >>>>>>>>>> consensus is what partly led up to the subsequent discussions and >>>>>>>>>> call for >>>>>>>>>> a more focused approach to reach consensus. If we already have a >>>>>>>>>> consensus >>>>>>>>>> on the storage model (separate tables and views), I think we should >>>>>>>>>> take >>>>>>>>>> things further and have continued focused discussions on the specific >>>>>>>>>> metadata in the form of a PR. I have included all previous >>>>>>>>>> discussions >>>>>>>>>> including the original doc and issue as references in the PR >>>>>>>>>> description. >>>>>>>>>> Please let me know if this works. Happy to hear others' thoughts on >>>>>>>>>> the >>>>>>>>>> best way to move forward. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Walaa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, May 8, 2024 at 12:56 AM Jan Kaul >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Walaa for trying to move things along. However I don't >>>>>>>>>>> think it's a good idea to start a separate discussion about the >>>>>>>>>>> metadata >>>>>>>>>>> for materialized views because we already had this discussion and >>>>>>>>>>> reached >>>>>>>>>>> consensus in this google doc: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing >>>>>>>>>>> >>>>>>>>>>> Once the draft is finalized we can adopt the PR to reflect the >>>>>>>>>>> consensus from the google doc. >>>>>>>>>>> >>>>>>>>>>> Best wishes, >>>>>>>>>>> >>>>>>>>>>> Jan >>>>>>>>>>> On 07.05.24 19:11, Walaa Eldin Moustafa wrote: >>>>>>>>>>> >>>>>>>>>>> Thanks Steven. I feel it is needed so the MV spec is not >>>>>>>>>>> scattered across the table and view spec pages. We may add a >>>>>>>>>>> reference in >>>>>>>>>>> each respective properties section. >>>>>>>>>>> >>>>>>>>>>> On Tue, May 7, 2024 at 10:04 AM Steven Wu >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Walaa, thanks for initiating the next step. >>>>>>>>>>>> >>>>>>>>>>>> With the agreed model of separate view and storage table, I am >>>>>>>>>>>> wondering if a separate materialized view spec page is needed. >>>>>>>>>>>> E.g., the >>>>>>>>>>>> new view metadata (view-materialized and view-storage-table) is >>>>>>>>>>>> probably >>>>>>>>>>>> good to be added to the view page directly to avoid information >>>>>>>>>>>> scattering. >>>>>>>>>>>> The same can be said about the storage table metadata. >>>>>>>>>>>> >>>>>>>>>>>> We may keep the separate materialized view page to document >>>>>>>>>>>> motivation, freshness semantics, etc.. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 6, 2024 at 10:58 PM Walaa Eldin Moustafa < >>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks again for participating in the modeling discussion [1]. >>>>>>>>>>>>> Since the outcome of this discussion was to model materialized >>>>>>>>>>>>> views as >>>>>>>>>>>>> separate objects, an Iceberg view and a table, I think the next >>>>>>>>>>>>> step >>>>>>>>>>>>> should be discussing the metadata details for each object. I have >>>>>>>>>>>>> created a >>>>>>>>>>>>> PR https://github.com/apache/iceberg/pull/10280 with an >>>>>>>>>>>>> initial spec improvement. Please feel free to review it and leave >>>>>>>>>>>>> feedback >>>>>>>>>>>>> there. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://lists.apache.org/thread/rotmqzmwk5jrcsyxhzjhrvcjs5v3yjcc >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Walaa. >>>>>>>>>>>>> >>>>>>>>>>>>> -- John Zhuge

Re: [Proposal] Add support for Materialized Views in Iceberg

>>> >>>> Thanks, >>>> Walaa. >>>> >>>> [1] https://github.com/apache/iceberg/issues/10043 >>>> [2] https://github.com/apache/iceberg/pull/9830 >>>> [3] >>>> https://docs.google.com/document/d/1zg0wQ5bVKTckf7-K_cdwF4mlRi6sixLcyEh6jErpGYY >>>> [4] https://github.com/apache/iceberg/issues/6420 >>>> [5] >>>> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF >>>> [6] https://lists.apache.org/thread/tb3wcs7czjvjbq9y1qtr87g9s95ky5zh >>>> [7] https://lists.apache.org/thread/l6cvrp4r1001k08cy2ypybzy2kgxpt1y >>>> >>> -- John Zhuge

Re: [ANNOUNCE] Apache Iceberg release 1.5.0

-1.5.0/apache-iceberg-1.5.0.tar.gz >>> >>> Release notes: https://iceberg.apache.org/releases/#150-release >>> >>> Java artifacts are available from Maven Central. >>> >>> Thanks to everyone for contributing! >>> >>> - Ajantha >>> >> -- John Zhuge

Re: [ANNOUNCE] Release Apache Iceberg Rust 0.2.0

Xiaoyang Liu, Xuanwo, ZENOTME, barronw, >> hiirrxnn, y0psolo, yi wang, zhjwpku and of course dependabot[bot] for >> working on this first release! >> >> Thanks >> On behalf of Apache Iceberg Community >> > -- John Zhuge

Re: [DISCUSS] Iceberg community summit

gt; talks >>>>>> and guiding the process. Once we have a selection committee, we can put >>>>>> together the concrete proposal for the ASF and the Iceberg PMC to request >>>>>> the ability to use the name Iceberg. >>>>>> >>>>>> If you'd like to help and be part of the selection committee, please >>>>>> volunteer in a reply to this thread. >>>>>> >>>>>> Since we likely can't include everyone that volunteers, I propose >>>>>> that the PMC should choose the final committee from the set of people >>>>>> that >>>>>> volunteer. We'll leave this open for the next week or so to give people >>>>>> time. >>>>>> >>>>>> Ryan >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> >>>>> -- John Zhuge

Re: [PROPOSAL] Improvement on our PR flows

; >>>> This issue has been closed because it has not received any >>>> activity in the last 14 days >>>> since being marked as 'stale' >>>> stale-pr-message: 'This pull request has been marked as >>>> stale due to 15 days of inactivity. It will be closed in 1 week if no >>>> further activity occurs. If you think that’s incorrect or this pull >>>> request requires a review, please simply write any comment. If closed, >>>> you can revive the PR at any time and @mention a reviewer or discuss >>>> it on the dev@iceberg.apache.org list. Thank you for your >>>> contributions.' >>>> close-pr-message: 'This pull request has been closed due to >>>> lack of activity. If you think that is incorrect, or the pull request >>>> requires review, you can revive the PR at any time.' >>>> stale-pr-label: 'stale' >>>> days-before-pr-stale: 15 >>>> days-before-pr-close: 7 >>>> exempt-pr-labels: "pinned,security" >>>> operations-per-run: 100 >>>> >>>> Thoughts ? >>>> >>>> PS: I did set up this on Apache Beam for example, and we did speed up >>>> the review and PR flows. >>>> >>>> Regards >>>> JB >>>> >>> -- John Zhuge

Re: Column-Level Key-Value Properties (Tags) in Iceberg

include these >>> features in its APIs. For the sake of this thread we can focus on the >>> Iceberg APIs aspect. >>> >>> >>> Here are some references to similar concepts in other systems: >>> >>> >>> * Avro attributes: *Avro 1.10.2 Specification - Schemas* >>> <https://avro.apache.org/docs/1.10.2/spec.html#schemas> (see >>> "Attributes not defined in this document are permitted as metadata"). >>> >>> * BigQuery policy tags: *BigQuery Column-level Security* >>> <https://cloud.google.com/bigquery/docs/column-level-security#set_policy> >>> . >>> >>> * Snowflake object tagging: *Snowflake Object Tagging Documentation* >>> <https://docs.snowflake.com/en/user-guide/object-tagging#create-and-assign-tags> >>> (see references to "MODIFY COLUMN"). >>> >>> >>> Looking forward to your insights on whether addressing this issue at the >>> Iceberg specification and API level is a reasonable direction. >>> >>> >>> Thanks, >>> Walaa. >>> >>> >>> >>> -- John Zhuge

Re: [Discuss] Spark 3.2 support?

+1 John Zhuge On Thu, Dec 14, 2023 at 4:03 PM Jack Ye wrote: > +1 > > -Jack > > On Wed, Dec 13, 2023 at 11:25 PM Eduard Tudenhoefner > wrote: > >> +1 on removing Spark 3.2 >> >> On Wed, Dec 13, 2023 at 8:01 PM Jean-Baptiste Onofré >> wrote: &g

Re: [DISCUSSION] Rename master branch as main for the main repository

://www.apache.org/foundation/policies/conduct>) >>> >>> > > especially contains section 5 about the wording we use. Several >>> Apache >>> > > projects renamed the master branch to the main branch (Apache >>> Karaf, >>> > > ActiveMQ, Airflow, ...). >>> > > As we already use main for go, rust, and python repositories, I >>> wonder >>> > > (for consistency) if we should not rename master to main on the >>> "main" >>> > > repository. >>> > > >>> > > Apache INFRA can do this "smoothly" but we would have to do some >>> changes: >>> > > \- update build.gradle >>> > > \- update README.md >>> > > \- update to GH Actions (in .github/workflows/*) >>> > > >>> > > Thoughts ? >>> > > >>> > > Regards >>> > > JB >>> > > >>> > >>> > >>> >> -- John Zhuge

Re: Help needed with Iceberg on Databricks platform

Iceberg supported > features? Any help would be greatly appreciated. Thank you in advance. > > Best regards, > Pani > -- John Zhuge

Re: Support create table like for Iceberg table?

ion z". But we >> can't do this for the Iceberg table. >> >> If this is a feature that is missing, should we collaborate to build a >> similar feature? >> >> Thanks >> >> >> > > -- > Ryan Blue > Tabular > -- John Zhuge

Re: [DISCUSS] Dropping Spark 2.4 support

+1 on removing 2.4 support On Fri, Apr 14, 2023 at 5:31 PM John Zhuge wrote: > Netflix internal Spark 2.4 is different from OSS. It is closer to OSS 3.0 > or 3.1 because it has DataSourceV2 and catalog support. So we don't rely on > Iceberg Spark 2.4 code. > > On Fri, Apr

Re: [DISCUSS] Dropping Spark 2.4 support

gt; actively migrating to Spark-3.x and Iceberg 1.1 (or later). I do not >> anticipate us using Spark-2.4.4 with newer versions of Iceberg (>0.9). >> If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that >> should be ok. >> @John Zhuge can you please chime i

Re: [Discuss] Allow all users who have Committed to the project to run CI without Approval

ommitter busy work >>>>> required >>>>> to validate new pull requests. >>>>> >>>>> I've created a new Infra ticket >>>>> https://issues.apache.org/jira/browse/INFRA-24400 to change our >>>>> behavior back to the old standard. >>>>> >>>>> I'd like to make sure folks are generally in favor of changing the >>>>> default back, please respond to this thread if you are in support of >>>>> going back to "Only requires approval first time" or if you don't >>>>> believe this is a good idea please respond as well. >>>>> >>>>> >>>>> Thanks for your time, >>>>> Russ >>>> >>>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >> -- John Zhuge

Re: Current Status of View Specification

*[EXTERNAL] Current Status of View Specification >>> >>> >>> >>> *CAUTION*: This email originated from outside of the organization. Do >>> not click links or open attachments unless you can confirm the sender and >>> know the content is safe. >>> >>> >>> >>> Hi Iceberg Dev, >>> >>> I see the spec has been checked in but I couldn't find a vote thread >>> ratifying it as a final V1 version (I might have been using the wrong >>> search terms) but for other additions of things like puffin it seemed like >>> there was an official vote. >>> >>> >>> >>> Should the spec be considered finalized as a V1 version now? Was there >>> a vote held? Will there be one? Are there any blockers to adoption? >>> >>> >>> >>> Thanks, >>> >>> Micah >>> >> >> >> -- >> Ryan Blue >> Tabular >> > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 1.1.0 RC4

arball, signature, and checksums are here: >>>> * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.1.0-rc4 >>>> >>>> You can find the KEYS file here: >>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>>> >>>> Convenience binary artifacts are staged on Nexus. The Maven repository URL >>>> is: >>>> * https://repository.apache.org/content/repositories/orgapacheiceberg-1114/ >>>> >>>> Please download, verify, and test. >>>> >>>> Please vote in the next 72 hours. >>>> >>>> [ ] +1 Release this as Apache Iceberg 1.1.0 >>>> [ ] +0 >>>> [ ] -1 Do not release this because... >>>> >>>> -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.14.1 RC3

.org/content/repositories/orgapacheiceberg-1105/ >>>> >>>> The 0.14.1 milestone tracks the bugs that are fixed in this release: >>>> * https://github.com/apache/iceberg/milestone/21?closed=1 >>>> >>>> Notable fixes include: >>>> * #5683 - Core: Fix exception handling in BaseTaskWriter (Flink's >>>> double close problem) >>>> * #5437 - Core, AWS: Fix Kryo serialization failure for FileIO >>>> * #5681 - Parquet: Close zstd input stream early to avoid memory >>>> pressure >>>> * #5691 - Spark: Fix stats in rewrite metadata action after >>>> partitioning changes >>>> >>>> Please download, verify, and test. >>>> >>>> This vote will be open for the next 72 hours. >>>> >>>> [ ] +1 Release this as Apache Iceberg 0.14.1 >>>> [ ] +0 >>>> [ ] -1 Do not release this because... >>>> >>>> -- >>>> Ryan Blue >>>> >>> >>> >>> -- >>> Ryan Blue >>> >> >> -- John Zhuge

Re: Welcome Yufei Gu as a committer

the project. > > Thanks for all your hard work, Yufei! > > - Anton > > -- > > [image: Image removed by sender.] > Sam Redai > > Developer Advocate | Tabular <https://tabular.io/> > > > > > -- > > > > > > Regards, > > Sumeet Gajjar. > > *-- Torture the data, and it will confess to anything* > > > > > -- > > Ryan Blue > > Tabular > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.14.0 RC1

>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >> >> Convenience binary artifacts are staged in Nexus. The Maven repository >> URL is: >> * >> https://repository.apache.org/content/repositories/orgapacheiceberg-1099/ >> >> This release includes: >> >> * MERGE and UPDATE plans using v2 deletes in Spark >> >> * A REST catalog implementation >> >> * Support for Spark 3.3 >> >> * Support for Flink 1.15 >> >> * A format for stats, sketches, and indexes (Puffin) >> >> * Zorder support while rewriting data files >> >> * Ranged IO reads >> >> * Additional metrics collection >> >> >> Please download, verify, and test. >> >> Please vote in the next 72 hours. >> >> [ ] +1 Release this as Apache Iceberg 0.14.0 >> [ ] +0 >> [ ] -1 Do not release this because... >> >> >> >> -- >> >> Ryan Blue >> >> >> >> >> -- >> >> Ryan Blue >> >> Tabular >> >> >> >> >> -- >> >> Ryan Blue >> >> Tabular >> > -- John Zhuge

Re: [VOTE] Adopt Puffin format as a file format for statistics and indexes

n file format specification: >>>>>> https://github.com/apache/iceberg/blob/master/format/puffin-spec.md >>>>>> (previous discussions: https://github.com/apache/iceberg/pull/4944, >>>>>> https://github.com/apache/iceberg-docs/pull/69) >>>>>> >>>>>> Intend use: >>>>>> * statistics in Iceberg tables (see >>>>>> https://github.com/apache/iceberg/pull/4945 and associated proposed >>>>>> implementation https://github.com/apache/iceberg/pull/4741) >>>>>> * in the future: storage for secondary indexes >>>>>> >>>>>> Puffin file reader and writer implementation: >>>>>> https://github.com/apache/iceberg/pull/4537 >>>>>> >>>>>> Thanks, >>>>>> PF >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Ryan Blue >>>>> Tabular >>>>> >>>> >>> >>> -- >>> Best Regards >>> >> -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.13.2 RC1

t(HiveSessionImpl.java:510) >>>> at >>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267) >>>> at >>>> org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139) >>>> ... 1 more >>>> >>>> Thanks you for working on this, >>>> Kyle >>>> >>>> >>>> >>>> >>>> On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner >>>> wrote: >>>> >>>>> Hi Everyone, >>>>> >>>>> I propose that we release the following RC as the official Apache >>>>> Iceberg 0.13.2 release. >>>>> >>>>> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf* >>>>> >>>>> >>>>>- This corresponds to the tag: *apache-iceberg-0.13.2-rc1* >>>>>- >>>>>https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1 >>>>>- >>>>> >>>>> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf >>>>> >>>>> >>>>> The release tarball, signature, and checksums are here: >>>>> >>>>>- >>>>> >>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1 >>>>> >>>>> >>>>> You can find the KEYS file here: >>>>> >>>>>- https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>>>> >>>>> >>>>> Convenience binary artifacts are staged on Nexus. The Maven repository >>>>> URL is: >>>>> >>>>>- >>>>> >>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/ >>>>> >>>>> >>>>> Please download, verify, and test. >>>>> >>>>> Please vote in the next 72 hours. >>>>> >>>>> [ ] +1 Release this as Apache Iceberg >>>>> [ ] +0 >>>>> [ ] -1 Do not release this because... >>>>> >>>> -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.13.0 RC2

gt; Please download, verify, and test. > > Please vote in the next 72 hours. > > [ ] +1 Release this as Apache Iceberg 0.13.0 > [ ] +0 > [ ] -1 Do not release this because... > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.13.0 RC1

: FAILED open or read shasum: WARNING: 1 listed file could not be read On Thu, Jan 27, 2022 at 10:05 AM John Zhuge wrote: > After importing keys from > https://dist.apache.org/repos/dist/dev/iceberg/KEYS, failed to verify the > signature: > > [apache-iceberg-0.13.0-rc1]$ ls > K

Re: [VOTE] Release Apache Iceberg 0.13.0 RC1

>> >>> The release tarball, signature, and checksums are here: >>> * >>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.0-rc1 >>> >>> You can find the KEYS file here: >>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>> >>> Convenience binary artifacts are staged on Nexus. The Maven repository >>> URL is: >>> * >>> https://repository.apache.org/content/repositories/orgapacheiceberg-1079/ >>> >>> Please download, verify, and test. >>> >>> Please vote in the next 72 hours. >>> >>> [ ] +1 Release this as Apache Iceberg 0.13.0 >>> [ ] +0 >>> [ ] -1 Do not release this because... >>> >> -- John Zhuge

Re: Welcome new PMC members!

zing at reviewing and helping people in the >> community and the PMC has decided to invite them to join. Congratulations, >> Jack and Russell! Thank you for all your hard work and support for the >> project. >> >> Ryan >> >> -- >> Ryan Blue >> > -- > Tomer Shiran > Founder and CPO, Dremio > > > -- John Zhuge

Re: Help improve Iceberg community meeting experience

zones, and most meetings can only >>> satisfy 2 time zones at best. >>> >>> I have asked AWS internally but we can only offer free use of AWS Chime, >>> which is not a very popular choice and would probably result in fewer >>> people joining the meetings. >>> >>> Any thoughts around this area? >>> >>> Best, >>> Jack Ye >>> >>> >>> >>> -- John Zhuge

Re: Proposal: Support for views in Iceberg

t;>>>>>> Regarding SQL dialects.My personal opinion is: Enforcing >>>>>>>> ANSI-compliant SQL across all engines is hard and probably not >>>>>>>> desirable while storing Calcite makes it hard for eg python to use >

Re: Subject: [VOTE] Release Apache Iceberg 0.12.0 RC3

t;> >>> You can find the KEYS file here: >>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>> >>> Convenience binary artifacts are staged in Nexus. The Maven repository >>> URL is: >>> * >>> https://repository.apache.org/content/repositories/orgapacheiceberg-1018/ >>> >>> Please download, verify, and test. >>> >>> Please vote in the next 72 hours. >>> >>> [ ] +1 Release this as Apache Iceberg 0.12.0 >>> [ ] +0 >>> [ ] -1 Do not release this because... >>> >> -- John Zhuge

Re: Welcoming Jack Ye as a new committer!

Hi everyone, >>> >>> I'd like to welcome Jack Ye as a new Iceberg committer. >>> >>> Thanks for all your contributions, Jack! >>> >>> Ryan >>> >>> -- >>> Ryan Blue >>> >> -- John Zhuge

Re: Welcoming OpenInx as a new PMC member!

ue, Jun 29, 2021 at 1:52 PM Ryan Blue wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I'd like to welcome OpenInx (Zheng Hu) as a new Iceberg PMC member. >>>>> >>>>> Thanks for all your contributions and commitment to the >>>>> project, OpenInx! >>>>> >>>>> >>>>> Ryan >>>>> >>>>> -- >>>>> Ryan Blue >>>>> >>>> -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.11.1 RC0

t;>>> https://github.com/apache/iceberg/tree/29cf712a821aa937e176f2d79a5593c4a1429e7f >>>>> >>>>> The release tarball, signature, and checksums are here: >>>>> * >>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.1-rc0/ >>>>> >>>>> You can find the KEYS file here (make sure to import the new key that >>>>> was used to sign the release): >>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>>>> >>>>> Convenience binary artifacts are staged in Nexus. The Maven repository >>>>> URL is: >>>>> * >>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1016/ >>>>> >>>>> This patch release includes these fixes: >>>>> https://github.com/apache/iceberg/milestone/13?closed=1 >>>>> >>>>> Please download, verify, and test. >>>>> >>>>> Please vote in the next 72 hours. >>>>> >>>>> [ ] +1 Release this as Apache Iceberg 0.11.1 >>>>> [ ] +0 >>>>> [ ] -1 Do not release this because… >>>>> >>>>> Thanks, >>>>> Anton >>>>> >>>>> >>>>> >>>>> >> >> -- >> Edgar R >> > > -- John Zhuge

Re: Welcoming Russell Spitzer as a new committer

;>>>>>> >>>>>>>>>> Congratulations Russel! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Mar 29, 2021 at 9:10 AM Anton Okolnychyi < >>>>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>> Hey folks, >>>>>>>>>> >>>>>>>>>> I’d like to welcome Russell Spitzer as a new committer to the >>>>>>>>>> project! >>>>>>>>>> >>>>>>>>>> Thanks for all your contributions, Russell! >>>>>>>>>> >>>>>>>>>> - Anton >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fholdenkarau&data=04%7C01%7Cmiwang%40adobe.com%7Cc735846b3171442c7ea208d8f2cd78c9%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637526311598709087%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=grkvsDiZFP4amRKgpUz7IXIEFu%2Bfik4ZMPJJj%2B%2BUVV8%3D&reserved=0> >>>>>>>>>> >>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Famzn.to%2F2MaRAG9&data=04%7C01%7Cmiwang%40adobe.com%7Cc735846b3171442c7ea208d8f2cd78c9%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637526311598719080%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=QE9unvN0R08oxzLYGDXzwDXzDY4mZoUFPby7s0x4rbY%3D&reserved=0> >>>>>>>>>> >>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fuser%2Fholdenkarau&data=04%7C01%7Cmiwang%40adobe.com%7Cc735846b3171442c7ea208d8f2cd78c9%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637526311598719080%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=xSP47ADR36Y%2BSVFlcbUBUsdmzlaOVmLunMuGLUr7JTE%3D&reserved=0> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Ryan Blue >>>>>>>>> Software Engineer >>>>>>>>> Netflix >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards >>>>>>> >>>>>> >>> >>> -- >>> Edgar R >>> >> -- John Zhuge

Re: Basic iceberg metrics viz tool

ound > > the emergence of reset.css files). > > Let me know if you think this would make a good addition to the repo. > > > > [1] https://github.com/plotly/plotly.js/blob/master/LICENSE > > > > -- > > Filip Bocse > > > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.11.0 RC0

Ignore my last email. Vote already passed. On Tue, Jan 26, 2021 at 11:40 AM John Zhuge wrote: > Got 404 for these 2 links: > >- >https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.11.0-rc0 >- >https://repository.apache.org/content/repositories/org

Re: [VOTE] Release Apache Iceberg 0.11.0 RC0

Spark cache optional >>>>>- #1798 fixes read failure when encountering duplicate entries of >>>>>data files >>>>>- #1785 fixes invalidation of metadata tables in CachingCatalog >>>>>- #1784 fixes resolving of SparkSession table's metadata tables >>>>> >>>>> *Other notable changes* >>>>> >>>>>- NaN counter is added to format v2 metrics >>>>>- Shared catalog properties are added in core library to >>>>>standardize catalog level configurations >>>>>- Spark and Flink now supports dynamically loading customized >>>>>`Catalog` and `FileIO` implementations >>>>>- Spark now supports loading tables with file paths via >>>>>HadoopTables >>>>>- Spark 2 now supports loading tables from other catalogs, like >>>>>Spark 3 >>>>>- Spark 3 now supports catalog names in DataFrameReader when using >>>>>Iceberg as a format >>>>>- Hive now supports INSERT INTO, case insensitive query, >>>>>projection pushdown, create DDL with schema and auto type conversion >>>>>- ORC now supports reading tinyint, smallint, char, varchar types >>>>>- Hadoop catalog now supports role-based access of table listing >>>>> >>>>> Please download, verify, and test. >>>>> >>>>> Please vote in the next 72 hours. >>>>> >>>>> [ ] +1 Release this as Apache Iceberg 0.11.0 >>>>> [ ] +0 >>>>> [ ] -1 Do not release this because... >>>>> >>>>> >>>>> >>> >>> -- >>> Ryan Blue >>> Software Engineer >>> Netflix >>> >> -- John Zhuge

Re: Iceberg At Adobe

Very nice! On Thu, Dec 3, 2020 at 10:36 AM Miao Wang wrote: > Hi, > > > > Our team post 1 blog about Iceberg use case at Adobe. > > > > https://medium.com/adobetech/iceberg-at-adobe-88cf1950e866 > > > > There will be a series of blogs to show more details. > > > > Miao > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.10.0 RC5

epository >>>>> URL is: >>>>> * >>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1013 >>>>> >>>>> This release includes important changes: >>>>> >>>>> * Flink support >>>>> * Hive read support >>>>> * ORC support fixes and improvements >>>>> * Application of row-level delete files on read >>>>> * Snapshot partition summary >>>>> * Ability to load LocationProvider dynamically >>>>> * Sort spec >>>>> >>>>> Please download, verify, and test. >>>>> >>>>> Please vote in the next 72 hours. >>>>> >>>>> [ ] +1 Release this as Apache Iceberg 0.10.0 >>>>> [ ] +0 >>>>> [ ] -1 Do not release this because… >>>>> >>>>> Thanks, >>>>> Anton >>>>> >>>> >> > > -- > Ryan Blue > Software Engineer > Netflix > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.10.0 RC4

running: sha512sum -c >>>>>>>> apache-iceberg-xx.tar.gz.sha512 : OK >>>>>>>> 5. Untar the archive and go into the source directory: tar xzf >>>>>>>> apache-iceberg-xx.tar.gz && cd apache-iceberg-xx: OK >>>>>>>> 6. Run RAT checks to validate license headers: dev/check-license: OK >>>>>>>> 7. Build and test the project: ./gradlew build (use Java 8) : OK >>>>>>>> >>>>>>>> On Wed, Nov 4, 2020 at 8:25 AM Anton Okolnychyi < >>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>> >>>>>>>>> Hi everyone, >>>>>>>>> >>>>>>>>> I propose the following RC to be released as official Apache >>>>>>>>> Iceberg 0.10.0 release. >>>>>>>>> >>>>>>>>> The commit id is d39fad00b7dded98121368309f381473ec21e85f >>>>>>>>> * This corresponds to the tag: apache-iceberg-0.10.0-rc4 >>>>>>>>> * >>>>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.10.0-rc4 >>>>>>>>> * >>>>>>>>> https://github.com/apache/iceberg/tree/d39fad00b7dded98121368309f381473ec21e85f >>>>>>>>> >>>>>>>>> The release tarball, signature, and checksums are here: >>>>>>>>> * >>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.10.0-rc4/ >>>>>>>>> >>>>>>>>> You can find the KEYS file here (make sure to import the new key >>>>>>>>> that was used to sign the release): >>>>>>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>>>>>>>> >>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven >>>>>>>>> repository URL is: >>>>>>>>> * >>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1012 >>>>>>>>> >>>>>>>>> This release includes important changes: >>>>>>>>> >>>>>>>>> * Flink support >>>>>>>>> * Hive read support >>>>>>>>> * ORC support fixes and improvements >>>>>>>>> * Application of row-level delete files on read >>>>>>>>> * Snapshot partition summary >>>>>>>>> * Ability to load LocationProvider dynamically >>>>>>>>> * Sort spec >>>>>>>>> >>>>>>>>> Please download, verify, and test. >>>>>>>>> >>>>>>>>> Please vote in the next 72 hours. >>>>>>>>> >>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.10.0 >>>>>>>>> [ ] +0 >>>>>>>>> [ ] -1 Do not release this because… >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Anton >>>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Software Engineer >>>>>> Netflix >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best, Jingsong Lee >>>>> >>>> >> >> -- >> Edgar R >> >> >> -- John Zhuge

Re: [DISCUSS] Rename iceberg-hive module?

t;>>>> misleading to name the Hive runtime module iceberg-hive-runtime. It was >>>>>>>> also pointed out that the iceberg-hive module is confusing for other >>>>>>>> reasons: someone unfamiliar with it would expect to use it to work with >>>>>>>> Hive, but it has no InputFormat or StorageHandler classes. >>>>>>>> >>>>>>>> Both problems are a result of a poor name for iceberg-hive. Maybe >>>>>>>> we should rename iceberg-hive to iceberg-hive-metastore. >>>>>>>> >>>>>>>> The drawback is that a module people could use will disappear (I'm >>>>>>>> assuming we won't rename iceberg-mr to iceberg-hive right away). But >>>>>>>> most >>>>>>>> people probably use a runtime Jar, so it might be a good time to make >>>>>>>> this >>>>>>>> change before there are more people depending on it. >>>>>>>> >>>>>>>> What does everyone think? Should we do the rename? >>>>>>>> >>>>>>>> rb >>>>>>>> >>>>>>>> -- >>>>>>>> Ryan Blue >>>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Best, Jingsong Lee >>>>> >>>> > > -- > Best Regards > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.9.1 RC0

/content/repositories/orgapacheiceberg-1009/ >>>>> >>>>> This is a patch release on top of 0.9.0 with several fixes: >>>>> * A correctness fix for ORC timestamps before 1 Jan 1970 that were >>>>> written by Spark >>>>> * A read fix for ORC decimals with precision less than 18 >>>>> * A fix to support push-down predicates with negated expressions >>>>> * Fixes for imports from other shaded Guava locations >>>>> >>>>> For the full list, see the 0.9.x branch: >>>>> https://github.com/apache/iceberg/commits/0.9.x >>>>> >>>>> Please download, verify, and test. >>>>> >>>>> Please vote in the next 72 hours. >>>>> >>>>> [ ] +1 Release this as Apache Iceberg 0.9.1 >>>>> [ ] +0 >>>>> [ ] -1 Do not release this because... >>>>> >>>>> >>>>> -- >>>>> Ryan Blue >>>>> >>>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- John Zhuge

Re: [VOTE] Graduate to a top-level project

t; * Owen O'Malley >>>>>> * Parth Brahmbhatt >>>>>> * Ratandeep Ratti >>>>>> * Ryan Blue >>>>>> >>>>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Ryan Blue be appointed to >>>>>> the office of Vice President, Apache Iceberg, to serve in accordance >>>>>> with and subject to the direction of the Board of Directors and the >>>>>> Bylaws of the Foundation until death, resignation, retirement, removal >>>>>> or disqualification, or until a successor is appointed; and be it >>>>>> further >>>>>> >>>>>> RESOLVED, that the Apache Iceberg Project be and hereby is tasked with >>>>>> the migration and rationalization of the Apache Incubator Iceberg >>>>>> podling; and be it further >>>>>> >>>>>> RESOLVED, that all responsibilities pertaining to the Apache Incubator >>>>>> Iceberg podling encumbered upon the Apache Incubator PMC are hereafter >>>>>> discharged. >>>>>> >>>>>> Please vote in the next 72 hours. >>>>>> >>>>>> [ ] +1 Petition the IPMC to graduate to top-level project >>>>>> [ ] +0 >>>>>> [ ] -1 Wait to graduate because . . . >>>>>> -- >>>>>> Ryan Blue >>>>>> >>>>> >>>> >>>> -- >>>> Thanks, >>>> Romin >>>> >>>> >>>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> >> >> -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.8.0-incubating RC2

t;>> * >>>>>>> https://github.com/apache/incubator-iceberg/commits/apache-iceberg-0.8.0-incubating-rc2 >>>>>>> * https://github.com/apache/incubator-iceberg/tree/8c05a2f5 >>>>>>> >>>>>>> The release tarball, signature, and checksums are here: >>>>>>> * >>>>>>> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc2/ >>>>>>> >>>>>>> You can find the KEYS file here: >>>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS >>>>>>> >>>>>>> Convenience binary artifacts are staged in Nexus. The Maven >>>>>>> repository URL is: >>>>>>> * >>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1006/ >>>>>>> >>>>>>> This release contains many bug fixes and several new features: >>>>>>> * Actions to remove orphaned files and to optimize metadata for >>>>>>> query performance >>>>>>> * Support for ORC data files >>>>>>> * Snapshot cherry-picking >>>>>>> * Incremental scan planning based on table history >>>>>>> * In and notIn expressions >>>>>>> * An InputFormat for writing MR jobs >>>>>>> >>>>>>> Please download, verify, and test. >>>>>>> >>>>>>> Please vote in the next 72 hours. >>>>>>> >>>>>>> [ ] +1 Release this as Apache Iceberg 0.8.0-incubating >>>>>>> [ ] +0 >>>>>>> [ ] -1 Do not release this because... >>>>>>> >>>>>>> -- >>>>>>> Ryan Blue >>>>>>> >>>>>> >>>>> -- John Zhuge

Re: Shall we start a regular community sync up?

bit of a > lurker, but this roadmap is important to mine and I have a daily at 5pm :(. > >>>>>> > >>>>>>-Justin > >>>>>> > >>>>>>> On Mar 18, 2020, at 6:43 PM, Saisai Shao > wrote: > >>>>>>> > >>>>>>> 5pm PST in any day works for me. > >>>>>>> > >>>>>>> Looking forward to it. > >>>>>>> > >>>>>>> Thanks > >>>>>>> Saisai > >>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> 李响 Xiang Li > >>>> > >>>> 手机 cellphone ：+86-136-8113-8972 > >>>> 邮件 e-mail ：wate...@gmail.com > >> > >> > >> > >> -- > >> Best Regards > -- John Zhuge

Re: Iceberg articles for you

06137394821e82ecd63e48a88212aecffbccc9a0c4979360&ascene=1&uin=MTA5OTczNzM4MA%3D%3D&devicetype=Windows+10&version=62080079&lang=zh_CN&exportkey=AQFOS%2Fy8vF656%2FngawYuD6Q%3D&pass_ticket=C%2FCFUbdg5csPZZtKEXhVPjbxqMhsToJOr4XtmbqYV%2BV6HOUyqNdiz%2FO2OTD1aEim> >>> >>> The plan is to post a series of articles about Iceberg. If you also have >>> some ideas that want to promote or share, feel free to contact me. I could >>> help to edit and post. >>> -- >>> Best Regards >>> >> > > -- > >李响 Xiang Li > > 手机 cellphone ：+86-136-8113-8972 > 邮件 e-mail ：wate...@gmail.com > -- John Zhuge

Re: Welcome new committer and PPMC member Ratandeep Ratti

gt;>>>>> 李响 于2020年2月17日周一 上午11:05写道： >>>>>>> >>>>>>> CONGRATULATIONS!!! >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 17, 2020 at 9:50 AM Junjie Chen < >>>>>>> chenjunjied...@gmail.com> wrote: >>>>>>> >>>>>>> Congratulations! >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 17, 2020 at 5:48 AM Ryan Blue wrote: >>>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> >>>>>>> >>>>>>> I'd like to congratulate Ratandeep Ratti, who was just invited to >>>>>>> join the Iceberg committers adn PPMC! >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks for your contributions and reviews, Ratandeep! >>>>>>> >>>>>>> >>>>>>> >>>>>>> rb >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Ryan Blue >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Best Regards >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>>李响 Xiang Li >>>>>>> >>>>>>> 手机 cellphone ：+86-136-8113-8972 >>>>>>> 邮件 e-mail ：wate...@gmail.com >>>>>>> >>>>>>> >> -- John Zhuge

Re: Iceberg in Spark 3.0.0

to branch and get ready for Spark 3.0.0. >>>>>> Right now, I'm focused on getting a release out, but I can review patches >>>>>> for Spark 3.0. >>>>>> >>>>>> Anyone know if there are nightly builds of Spark 3.0 to test with? >>>>>> >>>>> >>>>> Seems like there're nightly snapshots built in >>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-sql_2.12/3.0.0-SNAPSHOT/ >>>>> - >>>>> I've started setting something up with these snapshots so I can probably >>>>> start working on this. >>>>> >>>>> Thanks! >>>>> >>>>> Cheers, >>>>> -- >>>>> Edgar Rodriguez >>>>> >>>>> >>>>> >>>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > > -- John Zhuge

Re: [ANNOUNCE] Apache Iceberg release 0.7.0-incubating

-0.7.0-incubating.jar>. >> To use Iceberg in Spark 2.4, add the runtime Jar to the jars folder of your >> Spark install. >> >> Additional information is available at >> http://iceberg.apache.org/releases/ >> >> Thanks to everyone that contributed to this release! This is the first >> Apache release of Iceberg! >> >> >> -- >> Ryan Blue >> >> >> -- >> Ryan Blue >> > > > -- John Zhuge

Re: [VOTE] Release Apache Iceberg 0.7.0-incubating RC4

+1 - Passed all 7 steps of release validation - Integrated into downstream Spark 2.3 and 2.1 branches and passed integration tests On Fri, Oct 18, 2019 at 5:14 PM Ryan Blue wrote: > Hi everyone, > > I propose the following RC to be released as official Apache Iceberg > 0.7.0-incubating release.

Re: [VOTE] Release Apache Iceberg 0.7.0-incubating RC1

Ryan fixed the kryo serialization issue in https://github.com/apache/incubator-iceberg/pull/546. On Mon, Oct 14, 2019 at 9:35 AM John Zhuge wrote: > - Passed all 7 steps > - Build source code at tag apache-iceberg-0.7.0-incubating-rc1 locally, > unit tests passed. However, my downstr

Re: [VOTE] Release Apache Iceberg 0.7.0-incubating RC1

$TaskCommit) On Mon, Oct 14, 2019 at 9:35 AM John Zhuge wrote: > - Passed all 7 steps > - Build source code at tag apache-iceberg-0.7.0-incubating-rc1 locally, > unit tests passed. However, my downstream Spark 2.3 branch failed > integration tests, possibly due to > https://github.com/a

Re: [VOTE] Release Apache Iceberg 0.7.0-incubating RC1

gt;>> name 'stagedIceberg' >> >>>>> url ' >> https://repository.apache.org/content/repositories/orgapacheiceberg-1000/ >> ' >> >>>>> } >> >>>>> } >> >>>>> >> >>>>> ext { >> >>>>> icebergVersion = '0.7.0-incubating' >> >>>>> } >> >>>>> >> >>>>> Then run the downstream project’s tests. >> >>>>> >> >>>>> Thanks for voting, everyone! >> >>>>> >> >>>>> rb >> >>>>> >> >>>>> >> >>>>> On Fri, Oct 11, 2019 at 6:18 PM Ryan Blue wrote: >> >>>>>> >> >>>>>> Hi everyone, >> >>>>>> >> >>>>>> I propose the following RC to be released as official Apache >> Iceberg 0.7.0-incubating release. >> >>>>>> >> >>>>>> The commit id is 028a8d0e65d9c713b9b040c592fa10641b6c867b >> >>>>>> * This corresponds to the tag: apache-iceberg-0.7.0-incubating-rc1 >> >>>>>> * >> https://github.com/apache/incubator-iceberg/tree/apache-iceberg-0.7.0-incubating-rc1 >> >>>>>> * >> https://github.com/apache/incubator-iceberg/tree/028a8d0e65d9c713b9b040c592fa10641b6c867b >> >>>>>> >> >>>>>> The release tarball, signature, and checksums are here: >> >>>>>> * >> https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.7.0-incubating-rc1/ >> >>>>>> >> >>>>>> You can find the KEYS file here: >> >>>>>> * https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS >> >>>>>> >> >>>>>> Convenience binary artifacts are staged in Nexus. The Maven >> repository URL is: >> >>>>>> * >> https://repository.apache.org/content/repositories/orgapacheiceberg-1001/ >> >>>>>> >> >>>>>> This is the first Apache Iceberg release. >> >>>>>> >> >>>>>> Please download, verify, and test; then vote in the next 72 hours. >> >>>>>> >> >>>>>> [ ] +1 Release this as Apache Parquet 0.7.0-incubating >> >>>>>> [ ] +0 >> >>>>>> [ ] -1 Do not release this because... >> >>>>>> >> >>>>>> -- >> >>>>>> Ryan Blue >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Ryan Blue >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> -- >> >>> Ryan Blue >> >>> Software Engineer >> >>> Netflix >> >>> >> >>> >> >> >> >> >> >> -- >> >> Ryan Blue >> >> Software Engineer >> >> Netflix >> >> >> >> -- >> Thanks & Best Regards >> > -- John Zhuge

Re: New committer and PPMC member, Anton Okolnychyi

gt;>>> >>>>>> I'd like to congratulate Anton Okolnychyi, who was just invited to >>>>>> join the Iceberg committers and PPMC! >>>>>> >>>>>> Thanks for all your contributions, Anton! >>>>>> >>>>>> rb >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> >>>>> >>>> >>>> -- >>>> Edgar Rodriguez >>>> >>> -- John Zhuge

Re: [VOTE] Add the python implementation

y have visibility, until we have a reason to move it > elsewhere. > > > > Please vote in the next 72 hours: > > > > [ ] +1: Commit the current Python PR implementation > > [ ] +0: . . . > > [ ] -1: Do not add the current implementation because . . . > > > > Thanks! > > > > rb > > > > -- > > Ryan Blue > > > > > -- > > Ryan Blue > > -- John Zhuge

Re: [VOTE] Community code reviews

uests > [ ] +0, . . . > [ ] -1, I think this is a bad idea because . . . > > > rb > > -- > Ryan Blue > Software Engineer > Netflix > -- John Zhuge