Re: Storing catalog directly on object store

2024-11-26 Thread Jack Ye
discussion after knowing all the new S3 features that might help this domain :) Best, Jack Ye On Tue, Nov 26, 2024 at 9:36 AM Nikhil Benesch wrote: > Hi all, > > With Amazon S3 announcing support for the If-Match header yesterday [0], > all the > major object store implementation

Re: [VOTE] Add Variant type to Iceberg Spec

2024-11-26 Thread Jack Ye
I am +1 on adding it to the spec and not waiting for Parquet. It feels like a better 2-way door decision compared to being blocked by Parquet ratification timeline. -Jack On Tue, Nov 26, 2024 at 10:05 AM Micah Kornfield wrote: > 2. We aren't going to formally close V3 Spec yet, so if we do end

Re: [DISCUSS] Apache Iceberg Summit 2025 - Selection Committee

2024-11-26 Thread Jack Ye
Thanks JB for starting this, excited about the next summit! I would also love to volunteer. -Jack On Tue, Nov 26, 2024 at 10:47 AM Sung Yun wrote: > Hi JB, thank you very much for starting this thread! > > I'd love to volunteer as well. > > Sung > > On 2024/11/26 18:22:13 Yufei Gu wrote: > > S

Re: [VOTE] Deprecate and remove last-column-id

2024-11-20 Thread Jack Ye
+1 -Jack On Tue, Nov 19, 2024 at 7:45 AM Russell Spitzer wrote: > +1 > > On Tue, Nov 19, 2024 at 4:11 AM Fokko Driesprong wrote: > >> Hey Manu, >> >> That's an excellent question. I took the following rationale: >> >>- For the code, the iceberg-core module, a minor release deprecation >>

Changing Ownership and Cadence for Catalog Community Sync

2024-11-19 Thread Jack Ye
we have on the discussion schedule. Please let us know if there are any thoughts about this proposal, and we can coordinate accordingly. Best, Jack Ye

Re: [DISCUSS] - Deprecate Equality Deletes

2024-11-19 Thread Jack Ye
in at scan level. Overall, I think we have a lot of options on the table to solve CDC in Iceberg for both read and write. Given the row lineage feature is fundamentally in conflict with the equality deletes, I would +1 for dropping equality delete support. Best, Jack Ye [1] https://github.co

Re: [VOTE] Release Apache Iceberg 1.7.0 RC1

2024-11-06 Thread Jack Ye
+1 (binding) - Verified signature, checksum, license - Ran build and test with JDK 11 and 17 - Ran AWS integration tests - Ran on Spark 3.5 with some manual tests Best, Jack Ye On Wed, Nov 6, 2024 at 9:01 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > +1 binding > > Veri

Re: [VOTE] Endpoint for refreshing vended credentials

2024-10-22 Thread Jack Ye
+1 (binding) Best, Jack Ye On Tue, Oct 22, 2024 at 9:32 AM Dmitri Bourlatchkov wrote: > Thanks for the reply Eduard! > > I think it is fine to defer fine-tuning credential refreshes to a later PR. > > I'm upgrading my vote to +1 (non-binding). > > Cheers, > Dmitri

Re: [VOTE] Standardize vended credentials in OpenAPI spec

2024-10-17 Thread Jack Ye
+1 (binding) Best, Jack Ye On Thu, Oct 17, 2024 at 3:05 PM Dmitri Bourlatchkov wrote: > +1 (non-binding) > > Cheers, > Dmitri. > > On Tue, Oct 15, 2024 at 1:15 PM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> Hey everyone, >> >>

Re: [VOTE] Table V3 Spec: Row Lineage

2024-10-10 Thread Jack Ye
+1, overall agree that we should add this! Best, Jack Ye On Thu, Oct 10, 2024 at 1:43 PM Daniel Weeks wrote: > +1 > > Thanks Russell! > > On Thu, Oct 10, 2024 at 6:57 AM Eduard Tudenhöfner < > etudenhoef...@apache.org> wrote: > >> I left a few comments on the p

Re: [DISCUSS] Defining a concept of "externally owned" tables in the REST spec

2024-10-10 Thread Jack Ye
ayer, and cache eviction could be improved with notifications. But that feels like an optimization. What is the value for having both approaches at the same time? Best, Jack Ye [1] https://lists.apache.org/thread/ohqfvhf4wofzkhrvff1lxl58blh432o6 On Wed, Oct 9, 2024 at 5:51 PM Dennis Huo wrote:

Re: [DISCUSS] REST: Refreshing vended credentials

2024-10-10 Thread Jack Ye
ds to fetch only the credentials independently. Such an API becomes handy to do improvements like caching. Best, Jack Ye [1] https://docs.aws.amazon.com/cli/latest/reference/lakeformation/get-temporary-glue-table-credentials.html On Thu, Oct 10, 2024 at 3:47 AM Eduard Tudenhöfner wrote: > Hey

Re: [DISCUSS] Iceberg Summit 2025 ?

2024-10-02 Thread Jack Ye
officially kick start the summit preparation process. Best, Jack Ye On Wed, Oct 2, 2024 at 10:06 AM Robert Stupp wrote: > This is great! > > I'd prefer a hybrid event - so +1 on making Iceberg Summit 2025 happen! > > Maybe US East Coast? > > Topics: I think, a good a

[Notice] Update to catalog sync meeting timezone 2

2024-09-24 Thread Jack Ye
to discuss, feel free to raise it in the devlist, or I can also help with coordinating a specific meeting for discussion at a different time. Looking forward to seeing everyone! Best, Jack Ye

Re: [VOTE] Merge REST Spec Change To Add New Scan Planning APIs

2024-09-03 Thread Jack Ye
before voting, to make sure people are aware of the latest design and address any additional comments? Best, Jack Ye [1] https://lists.apache.org/thread/qq13468x6gk0vxnsckzc5xd02tjlvpkm On Mon, Sep 2, 2024 at 9:22 PM Chertara, Rahil wrote: > Hi all, > > > > I've opened a PR

Re: [VOTE] Merge REST Spec change to add RemovePartitionSpecsUpdate update type

2024-08-28 Thread Jack Ye
+1 (binding) On Tue, Aug 27, 2024 at 5:21 AM roryqi wrote: > +1 > > Manu Zhang 于2024年8月27日周二 11:44写道: > >> +1 (non-binding) >> >> On Tue, Aug 27, 2024 at 11:00 AM xianjin wrote: >> >>> +1 (non-binding) >>> Sent from my iPhone >>> >>> On Aug 27, 2024, at 4:22 AM, Fokko Driesprong wrote: >>> >>

Re: [VOTE] REST Endpoint discovery

2024-08-20 Thread Jack Ye
+1 On Tue, Aug 20, 2024 at 12:37 PM Russell Spitzer wrote: > +1 > > On Tue, Aug 20, 2024 at 2:32 PM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> +1 non-biding >> >> Thanks for driving this Eduard. >> >> On Tue, Aug 20, 2024 at 12:17 PM Daniel Weeks wrote: >> >>> +1 >>> >>> On Tue

Re: [DISCUSS] REST Endpoint discovery

2024-08-15 Thread Jack Ye
gt;>> >>> On Thu, Aug 15, 2024 at 11:32 AM Eduard Tudenhöfner < >>> etudenhoef...@apache.org> wrote: >>> >>>> Hey Jack, >>>> >>>> thanks for the feedback. I replied in the doc but I can reiterate my >>>> answer here

Re: [DISCUSS] REST Endpoint discovery

2024-08-15 Thread Jack Ye
e required in the Iceberg spec, and fix it to be consistent, assuming nobody is taking a dependency on these operationIds right now. Personally speaking, I am pretty neutral on this topic, but curious what everyone thinks. Best, Jack Ye On Wed, Aug 14, 2024 at 9:20 AM Eduard Tudenhöfner wro

Re: Welcome Péter, Amogh and Eduard to the Apache Iceberg PMC

2024-08-14 Thread Jack Ye
Congratulations! On Wed, Aug 14, 2024 at 8:12 AM Amogh Jahagirdar <2am...@gmail.com> wrote: > Thanks all, and congrats to Péter and Eduard! > > On Wed, Aug 14, 2024 at 6:40 AM Steve Loughran > wrote: > >> >> congratulations all. >> >> On Tue, 13 Aug 2024 at 21:25, Russell Spitzer >> wrote: >> >

Re: [VOTE] Merge REST spec clarification on how servers should handle unknown updates/requirements

2024-08-14 Thread Jack Ye
+1 -Jack On Wed, Aug 14, 2024 at 8:39 AM Daniel Weeks wrote: > +1 > > On Tue, Aug 13, 2024 at 7:34 PM xianjin wrote: > >> +1 >> >> On Aug 14, 2024, at 2:24 AM, Ryan Blue >> wrote: >> >>  >> +1 >> >> On Tue, Aug 13, 2024 at 8:59 AM Yufei Gu wrote: >> >>> +1 >>> Yufei >>> >>> >>> On Tue, Aug

Re: [DISCUSS] Variant Spec Location

2024-08-14 Thread Jack Ye
+1 for copying the spec into our repository, I think we need to own it fully as a part of the table spec, and we can build compatibility through tests. -Jack On Wed, Aug 14, 2024 at 12:52 PM Russell Spitzer wrote: > I'm not really in favor of linking and annotating as that just makes > things m

Re: [DISCUSS] Changing namespace separator in REST spec

2024-08-07 Thread Jack Ye
Sorry a bit late to this thread. I would personally prefer the client side separator solution (query param with `?delim=.`) a bit more than the server side (config override), just given the experience of handling similar situations for Glue data catalog which allows any name for database (namespac

Re: [DISCUSS] adoption of format version 3

2024-08-02 Thread Jack Ye
ot;, which is true because you need to spend additional effort and release it. And without a reference implementation, it is hard to say if the spec is mature enough to be released, which again makes it potentially tied to the release cycle of at least the Java library. Curious what people think. Best

Re: [DISCUSS] Use iceberg-rust as pyiceberg file io

2024-08-02 Thread Jack Ye
+1 for an OpenDALFileIO -Jack On Fri, Aug 2, 2024 at 8:32 AM Xuanwo wrote: > Hi, renjie > > Thank you for your support. I'll delve into the details and first build a > PoC PR to make it clear. > > On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote: > > Hi: > > Thanks Xuanwo for raising this. > > A

Re: [VOTE] Clarify "File System Tables" in the table spec

2024-08-01 Thread Jack Ye
+1 (binding) -Jack On Thu, Aug 1, 2024 at 6:30 AM Russell Spitzer wrote: > +1 (Binding) > > On Thu, Aug 1, 2024 at 7:31 AM Fokko Driesprong wrote: > >> +1 (binding) >> >> Op do 1 aug 2024 om 09:57 schreef Eduard Tudenhöfner < >> etudenhoef...@apache.org>: >> >>> +1 (non-binding) >>> >>> On Thu

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-31 Thread Jack Ye
e, etc. that are not strictly portable across platforms? (2) Again, is it worth it for the community to maintain such an implementation? Curious what people think. -Jack [1] https://github.com/delta-io/delta/blob/master/PROTOCOL.md#last-checkpoint-file On Wed, Jul 31, 2024 at 8:05 AM Jack Ye wro

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-31 Thread Jack Ye
rename-org.apache.hadoop.fs.Path-org.apache.hadoop.fs.Path-org.apache.hadoop.fs.Options.Rename...- On Wed, Jul 31, 2024 at 8:04 AM Russell Spitzer wrote: > My guess would be to avoid complications with multiple committers > attempting to swap at the same time. > > On Wed, Jul 31, 2024 at 9:50

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-31 Thread Jack Ye
referenced in the hint file. These can go out of sync since the operation > there is not atomic. Removing this introduces other problems where you have > to determine the latest version of the metadata using prefix-listing, which > is not efficient and desirable on an object store as you alread

Re: [DISCUSS] Describing REST Server capabilities

2024-07-31 Thread Jack Ye
One thing to clarify, regarding per-endpoint versioning, my understanding is that endpoint version should bump (e.g. GET /v1/namespaces to GET /v2/namespaces) when there is a significant backwards incompatible change. -Jack On Tue, Jul 30, 2024 at 7:56 PM Jack Ye wrote: > > are you t

Re: [ANNOUNCE] Apache PyIceberg release 0.7.0

2024-07-30 Thread Jack Ye
Thank you Sung for managing the release! And many thanks to everyone that participated! Best, Jack Ye On Tue, Jul 30, 2024 at 12:24 PM Kevin Liu wrote: > Woot! Thank you, Sung, for managing the release! I'm very excited about > this new version. > > I want to highlight the

Re: [DISCUSS] Describing REST Server capabilities

2024-07-30 Thread Jack Ye
atalog version bump? > > On Tue, Jul 30, 2024 at 8:34 AM Jack Ye wrote: > >> Since the catalog sync was canceled this week, I find maybe it is better >> to reply here for my latest take on this topic. >> >> I think we have 2 discussions intertwined here, that

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-30 Thread Jack Ye
Atomicity is just one requirement, and it also needs to be efficient, desirably a metadata-only operation. Looking at some documentations of GCS [1], the rename operation is still a COPY + DELETE behind the scene unless it is a hierarchical namespace bucket. The Azure documentation [2] also sugges

Re: [DISCUSS] Describing REST Server capabilities

2024-07-30 Thread Jack Ye
curity perspective. >>>>>> >>>>>> Let's keep that capability flag. >>>>>> >>>>>> Cheers, >>>>>> Dmitri. >>>>>> >>>>>> On Wed, Jul 10, 2024 at 5:48 AM Eduard Tudenhöfner &l

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-30 Thread Jack Ye
fferent >>>> project that uses the Iceberg metadata format. >>>> >>>> On Tue, Jul 23, 2024 at 5:57 PM lisoda wrote: >>>> >>>>> >>>>> Sir, regarding this point, we have some experience. In my view, as >>>>>

Re: [VOTE] Release Apache PyIceberg 0.7.0rc2

2024-07-30 Thread Jack Ye
+1 (binding) - Verified signature, license, checksum - Ran build and tests (python 3.11) - Ran S3 and Glue integration and manual tests Best, Jack Ye On Tue, Jul 30, 2024 at 5:00 AM Mehul Batra wrote: > +1 (Non-binding) > >- Validated signatures/checksums/license >- Ran

Re: Meeting time for catalog community sync

2024-07-28 Thread Jack Ye
finalize the schedule, I propose the meetings going forward to use the following time and cadence: - Wednesday 9am pacific time, recurring every 3 weeks, starting 8/7 - Wednesday 8pm pacific time, recurring every 3 weeks, starting 8/14 What do we think? Best, Jack Ye On Sun, Jul 28, 2024 at 6

Re: [ANNOUNCE] Apache Iceberg release 1.6.0

2024-07-25 Thread Jack Ye
Thanks JB for organizing the release, and thanks everyone that has contributed! -Jack On Thu, Jul 25, 2024 at 8:07 AM Jean-Baptiste Onofré wrote: > The Apache Iceberg team is pleased to announce the release of Apache > Iceberg 1.6.0! > > Apache Iceberg is an open table format for huge analytic

Meeting time for catalog community sync

2024-07-24 Thread Jack Ye
te on this. I will go ahead to change the time directly if there is a clear preference here, let's see. Best, Jack Ye

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-24 Thread Jack Ye
te a new playlist of the videos posted > 3. We publish under the Apache Iceberg YouTube channel in a new playlist > > IMO #3 would be good so we can publish under the same roof and not have > community-led videos fractured over multiple accounts. Ryan, Jack, or > anyone else with some t

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-23 Thread Jack Ye
other catalog implementations still have value as either REST > back-end catalogs or as regular catalogs for many users. > > On Tue, Jul 23, 2024 at 9:11 AM Jack Ye wrote: > >> For some additional information, we also have some Iceberg HDFS users on >> EMR. Those are mainly u

Re: Dropping JDK 8 support

2024-07-23 Thread Jack Ye
eberg 2.0 plan). >>>>>>> >>>>>>> Also as Huaxin has discovered in Spark 4.0 Support PR >>>>>>> <https://github.com/apache/iceberg/pull/10622>, looks like we may >>>>>>> have to drop Java8 first in S

Re: Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-23 Thread Jack Ye
similar to iceberg-hadoop to extend >> hadoop_catalog? If it is removed, we won't be able to continue providing >> services to customers. So, if possible, please consider this option. >> >> Thank you all. >> >> Kind regards, >> lisoda >> >> >

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-23 Thread Jack Ye
c changes >>> 3. Define roles/responsibilities and provide guidance on how members >>> can grow into those roles. (Cover release manager, comitter, PMC member, >>> community member, with links as appropriate). >>> 4. Create guidelines on starting new revision

Re: [ANNOUNCE] Welcoming new committers and PMC members

2024-07-23 Thread Jack Ye
Congratulations!! Best, Jack Ye On Tue, Jul 23, 2024 at 8:16 AM Ryan Blue wrote: > Congratulations, everyone! And thanks for all your contributions! > > On Tue, Jul 23, 2024 at 8:06 AM Mehul Batra > wrote: > >> Congratulations Everyone! >> >> On Tue, Jul 23,

Re: [VOTE] Release Apache Iceberg 1.6.0 RC1

2024-07-22 Thread Jack Ye
+1 (binding) Checked signature, checksum, license Ran unit and integration tests with JDK17 Ran manual tests with Spark 3.5 Best, Jack Ye On Mon, Jul 22, 2024 at 3:02 PM Dmitri Bourlatchkov wrote: > +1 (nb) > > Verified the REST Client warning about OAuth2 URL (again, jus

Re: Dropping JDK 8 support

2024-07-22 Thread Jack Ye
+1 (binding), I did not expect this to be a vote thread, but overall +1 for dropping JDK8 support. -Jack On Mon, Jul 22, 2024 at 10:30 AM Yufei Gu wrote: > +1(binding), as much as I want to drop JDK 8, still encourage everyone to > spark out about any concerns. > Yufei > > > On Mon, Jul 22, 202

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-22 Thread Jack Ye
n [3] https://docs.google.com/document/d/1S3igb5NqSlYE3dq_qRsP3X2gwhe54fx-Sxq5hqyOe6I/edit [4] https://iceberg.apache.org/contribute/#how-are-proposals-adopted [5] https://orc.apache.org/develop/bylaws/ -Jack On Fri, Jul 19, 2024 at 2:29 PM Jack Ye wrote: > > specifically the discussion of the standar

Re: Building with JDK 21

2024-07-19 Thread Jack Ye
+1 for dropping JDK8 support and adding JDK21. > What does dropping Java 8 support mean to companies that are still using Java 8 for Iceberg in production? >From the AWS side, AWS Corretto JDK8 end of life is July 2026, see: https://aws.amazon.com/corretto/faqs/#support_calendar. I would suggest

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-19 Thread Jack Ye
ust is recognized by a PMC vote. > > Ryan > > On Fri, Jul 19, 2024 at 11:00 AM Owen O'Malley > wrote: > >> I meant specifically the discussion of the standard roles (eg. users, >> committers, pmc, pmc chair) that are well covered in >> https://www.apache.org

Re: [DISCUSS][BYLAWS] Moving forward on the bylaws

2024-07-19 Thread Jack Ye
was not intended to be tied to committer responsibilities. Best, Jack Ye [1] https://community.apache.org/newcommitter.html [2] https://www.apache.org/foundation/voting On Fri, Jul 19, 2024 at 10:22 AM Owen O'Malley wrote: > Everyone is welcome to vote. The Iceberg PMC will have the only

Re: [VOTE] Merge table spec clarifications on time travel and equality deletes

2024-07-19 Thread Jack Ye
+1 (binding) added minor comments to the time travel PR. Best, Jack Ye On Fri, Jul 19, 2024 at 8:22 AM Daniel Weeks wrote: > +1 (binding) > > Thanks, Micah. > > On Thu, Jul 18, 2024 at 8:29 PM Amogh Jahagirdar <2am...@gmail.com> wrote: > >> +1 (non-binding

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
t example. Do we consider playlists links? > I guess the question is whether it looks official. Would inclusion in a > playlist be mistaken for an endorsement? Or would we get into unnecessary > debates over what is appropriate for inclusion in a playlist? > > On Thu, Jul 18, 2024 at 3:39 

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
I don't think it is wise for the >>> project to decide what opinions can be posted there. >>> >>> I think that we should remove the last meetup video and only post >>> content like the community sync videos. For everything else, I think it is >>>

Re: Administration of Apache Iceberg Social/Marketing Channels

2024-07-18 Thread Jack Ye
> posting/content management privileges to the Iceberg YouTube channel for these meetup recordings This sounds reasonable to me, when the meetup is recurring. So I am good with doing this for the Seattle meetup series. We have technically already done so for Bits for the community sync meeting ser

Re: [DISCUSS] Deprecate HadoopTableOperations, move to tests in 2.0

2024-07-18 Thread Jack Ye
at. I could help raise a thread about it after this discussion is closed. Best, Jack Ye [1] https://cloud.google.com/storage/docs/object-versioning#file_restoration_behavior [2] https://cloud.google.com/storage/docs/xml-api/reference-headers#xgoogifgenerationmatch [3] https://learn.micr

Re: Iceberg namespace operation behaviors

2024-07-16 Thread Jack Ye
This is the public link: https://www.youtube.com/watch?v=JSN9T20eAuU -Jack On Tue, Jul 16, 2024 at 7:38 PM Manu Zhang wrote: > Hi Ajantha, > > It looks the catalog sync link is not publicly accessible. Can you check? > > Thanks, > Manu > > On Wed, Jul 17, 2024 at 12:42 AM Ajantha Bhat > wrote:

Re: [DISCUSS] Describing REST Server capabilities

2024-07-11 Thread Jack Ye
side instead of calling server and getting a "generic 501"? -Jack On Thu, Jul 11, 2024 at 9:51 AM Jack Ye wrote: > Sorry I will take a look at the new comments later today. > > -Jack > > On Wed, Jul 10, 2024, 11:42 PM Eduard Tudenhöfner < > etudenhoef...

Re: [DISCUSS] Describing REST Server capabilities

2024-07-11 Thread Jack Ye
ersioning information, clients can >>> make more informed decisions on which endpoints to call. One could argue >>> that generally throwing a 501 on everything that isn't supported might be >>> sufficient, but that doesn't necessarily help a cli

Re: [VOTE] spec: remove the JSON spec for content file and file scan task sections

2024-07-11 Thread Jack Ye
+1 (binding) On Thu, Jul 11, 2024 at 3:37 AM Piotr Findeisen wrote: > it looks it's part of the spec that's not connected to the other parts of > the spec (like "dead code") > > +1 (non binding) > > > On Thu, 11 Jul 2024 at 08:30, Eduard Tudenhöfner > wrote: > >> +1 (non-binding) >> >> On Thu,

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-11 Thread Jack Ye
Update: Based on the conversations on the incubator list ( https://lists.apache.org/thread/5ojxny76fr7n1y0hs2rxhr55g1fgcsln), I have updated the document title from "guidelines" back to "bylaws". Thanks for pointing this out Carl! I have also updated the document with a few suggested edits from s

Re: [DISCUSS] Enable the discussion tab for iceberg github repos

2024-07-10 Thread Jack Ye
+1 for enabling it on iceberg-rust first. I am curious how search-engine friendly this is, that will be a huge plus if people can find these discussion contents. For people who have used it more like Piotr, do you know about this? I search Trino things quite a lot on Google, but rarely did I find

Re: [VOTE] Fix property names in REST spec for statistics / partition statistics

2024-07-10 Thread Jack Ye
+1 (binding) -Jack On Wed, Jul 10, 2024 at 9:00 AM Steve Zhang wrote: > +1 (non binding) > > Thanks, > Steve Zhang > > > > On Jul 10, 2024, at 1:10 AM, Jean-Baptiste Onofré wrote: > > +1 (non binding) > > >

Re: [DISCUSS] Fix property names in REST spec for statistics / partition statistics

2024-07-09 Thread Jack Ye
vote to confirm. > Those can be separate things, like sending a [ANNOUNCE] message on this > list rather than having a [VOTE] thread open for 3 days before fixing it. > > Again, I'm asking a question about how we want to handle situations like > this moving forward. As I said original

Re: [DISCUSS] Fix property names in REST spec for statistics / partition statistics

2024-07-09 Thread Jack Ye
I agree with Robert here that we need to get into the habit of doing votes on the spec changes. There are typos that could be in sections like description, that does not affect the overall spec usage at all, maybe those changes do not need an official vote. However, this is a change of an existing

Re: [DISCUSS] Describing REST Server capabilities

2024-07-09 Thread Jack Ye
l 9, 2024 at 9:12 AM Dmitri Bourlatchkov > wrote: > >> Re: remote signing, I agree that it does not look like a server >> capability that a client can / should discover. It is more like something >> that the server instructs / configures the client to do. >> >

Re: [DISCUSS] Describing REST Server capabilities

2024-07-09 Thread Jack Ye
>>> v3 fields, and the server enforces the payload differently based on the >>> TableMetadata.format-version field. If the server does not support v3, it >>> can return unsupported at that time. >>> >>> Either way we go, the table-spec version does not nee

Re: [DISCUSS] Enable the discussion tab for iceberg github repos

2024-07-09 Thread Jack Ye
I am not familiar with the GitHub discussion feature, but could we start with GitHub Issue tags + templates to distinguish between actual issues vs this kind of questions? Why is that not sufficient? Also, if there are a lot of questions about the roadmap, I think we should discuss and make good m

Re: [Vote] Deprecate oauth tokens endpoint

2024-07-08 Thread Jack Ye
+1 (binding) There are some wording aspects that others still have comments in the PR, but in general, +1 for deprecating the endpoint. Best, Jack Ye On Mon, Jul 8, 2024 at 9:22 AM Robert Stupp wrote: > +1 > > On 08.07.24 18:15, Robert Stupp wrote: > > Hi Everyone, > >

Re: [Proposal] REST Spec: Server-side Metadata Tables

2024-07-03 Thread Jack Ye
d PlanTable APIs. Just a quick thought for now, I need to think a bit more about this. Best, Jack Ye On Wed, Jul 3, 2024 at 1:45 PM Yufei Gu wrote: > Hi folks, > > I'd like to discuss a new proposal to support server-side metadata tables. > > One of Iceberg's most ad

Re: [DISCUSSION] Addressing security questions in the Iceberg REST specification

2024-07-02 Thread Jack Ye
strict requirement. This can also serve > as documentation/reference of what each client (language/version) supports > and how. > > Robert > > > [1] https://projectnessie.org/nessie-latest/client_config/#oauth2-settings > > [2] > https://github.com/projectnessie/

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-02 Thread Jack Ye
Gmail and many people are still out of town, so things were also delayed at this front. Let us know what you think is the best way to proceed! Best, Jack Ye On Tue, Jul 2, 2024 at 9:14 AM Daniel Weeks wrote: > Thanks Owen, I really appreciate the offer to moderate the discussion. I >

Re: Support Securable Objects in Iceberg REST Catalog

2024-07-02 Thread Jack Ye
gt; > Eventually there's no way around "trust" between the engine and the > catalog. Establishing "trust" in a secure way is not that easy IMO. > > > On 02.07.24 06:30, Jack Ye wrote: > > Thanks Dennis for the detailed analysis and suggestions! Here are

Re: [DISCUSS] Describing REST Server capabilities

2024-07-02 Thread Jack Ye
Spec is an interesting topic we did not discuss. Robert, how do you envision this to be used? In my mind, if a new table format v3 is launched, there are 2 approaches we can go with, taking CreateTable as an example: (1) increment the related operation version, which means that POST /v2/{prefix}/

Re: Re: Support Securable Objects in Iceberg REST Catalog

2024-07-01 Thread Jack Ye
t; > catalog extension at LinkedIn using views [1], and one of the main > > improvements to that catalog extension would be securable view objects. > > Admittedly, it might require further improvements to compute engines to > > implement the permissions, but having an Iceberg spec would

Re: [DISCUSS] Describing REST Server capabilities

2024-07-01 Thread Jack Ye
occur. >>> >>> For new versions that are still in development, it's possibly the >>> easiest to not do anything special, not even announce the new version. >>> Whether we indicate some "beta version" or reserve "Integer.MAX_VALUE" >

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-07-01 Thread Jack Ye
Hi everyone, Thanks for all the comments and feedback on the document, I am working with a few commenters on some additional changes and wording, and then will carry out the vote. Best, Jack Ye On Thu, Jun 27, 2024 at 11:02 AM Jack Ye wrote: > To provide an update here, I have consolida

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-27 Thread Jack Ye
me know if there is anything else we see major disagreements with, and I will organize a vote after 24 hours. Best, Jack Ye On Wed, Jun 26, 2024 at 11:04 AM Jack Ye wrote: > +1 for adding to the site. > > I am putting it as a doc for now since Google doc is easier to comment (I >

Re: [DISCUSS] Describing REST Server capabilities

2024-06-27 Thread Jack Ye
> > On Thu, Jun 27, 2024 at 11:34 AM Jean-Baptiste Onofré > wrote: > >> Hi Jack >> >> I like Robert's proposal. Back to the topics, I think grouping with >> tags is more "flexible" (it was what we included in the REST spec >> proposal a

Re: [DISCUSS] Describing REST Server capabilities

2024-06-26 Thread Jack Ye
at's net better for the ecosystem since a broader set of users have > a clear idea of what's supported and can make good decisions for themselves > since everyone is speaking the same standard language. > > Furthermore, we could also look at if it makes sense to make the tags mo

Re: [Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-26 Thread Jack Ye
version first, and then we just go through all the identified topics one by one. I have a list of all topics in the original feedback collection devlist thread. Let me know what you think about these plans! Best, Jack Ye On Wed, Jun 26, 2024 at 9:04 AM Ryan Blue wrote: > +1 for adding this to

Re: [DISCUSS] Describing REST Server capabilities

2024-06-26 Thread Jack Ye
ctive. Best, Jack Ye On Wed, Jun 26, 2024 at 9:02 AM Daniel Weeks wrote: > I think Robert's approach is a reasonable compromise here. > > If we wanted a "per operation/endpoint" versioning, I think I'd prefer > Micah's OpenAPI spec based approach because i

Re: Iceberg Catalog Syncs Invite

2024-06-26 Thread Jack Ye
AM Jean-Baptiste Onofré > wrote: > >> Hi Jack >> >> Thanks ! >> Just a reminder: we have to provide regular updates on the dev mailing >> list (as we say at Apache: if it doesn't happen on the mailing list, >> it never happened). >> >> Regard

Re: Iceberg Catalog Syncs Invite

2024-06-25 Thread Jack Ye
The google group is purely for subscribing to the meeting. We do not plan to use it for discussions. -Jack On Tue, Jun 25, 2024, 10:33 PM Xuanwo wrote: > Hi, Jack Ye > > Thank you for setting up the Iceberg Catalog Sync; I'm eager to > participate. > > However, it feel

Iceberg Catalog Syncs Invite

2024-06-25 Thread Jack Ye
the sync meetings! Best, Jack Ye

Re: [DISCUSS] Describing REST Server capabilities

2024-06-25 Thread Jack Ye
bility, I think not returning anything should mean that all API that the client understands are having version *. What do people think of it, compared to the tag approach? Best, Jack Ye On Mon, Jun 24, 2024 at 1:42 PM Micah Kornfield wrote: > I don't have strong opi

[Discussion] Apache Iceberg Guidelines for Committership and PMC Membership

2024-06-25 Thread Jack Ye
, Jack Ye

Re: Feedback Collection: Bylaws in Iceberg

2024-06-25 Thread Jack Ye
topics one by one. Feel free to keep providing comments or feedback in this thread, and please let me know any additional topics I did not cover in the list above! Best, Jack Ye On Tue, Jun 25, 2024 at 8:54 AM Honah J. wrote: > Hi everyone, > > Thanks to everyone for the valuab

[Discussion] Apache Iceberg Community Guideline - Initial Version

2024-06-25 Thread Jack Ye
hing is probably to agree upon the 2/3 majority approval for modifying the project guidelines, so we can have a consistent voting method going forward. This initial introduction of the bylaws will be voted using consensus approval. Please take a look and comment about any additional changes needed, and I will host a vote in 3 days. Best, Jack Ye

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
conduct. And note that >> I'm one of the individuals here. >> >> Respectfully, what does this mean, Ryan? No individual was even singled >> out here. This comes off as stifling discussion, not cool... >> >> On Mon, Jun 24, 2024, 9:08 AM Jack Ye wrote: >&g

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
ing etc. These important contributions are not >>> making the expected progress. It might be helpful to create bylaws or >>> procedures to ensure these proposals and PRs receive the necessary >>> attention and are addressed promptly. This could involve setting timeframes >>&

Re: Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
n" in Apache is vague, and I don't think there is a requirement to say a proposal is a part of that and stick to the vague standard process. Maybe we should re-discuss this and formally conclude a direction with everyone. Best, Jack Ye On Mon, Jun 24, 2024 at 6:19 AM Renjie Liu wrote:

Feedback Collection: Bylaws in Iceberg

2024-06-24 Thread Jack Ye
orward to discussing this with everyone! Best, Jack Ye

Support Securable Objects in Iceberg REST Catalog

2024-05-30 Thread Jack Ye
implement for managing access to objects in IRC, and (2) users that are on one IRC product do not need to be locked-in due to access management aspects. Would really appreciate any feedback on this topic and proposal! Best, Jack Ye

Re: Addressing security questions in the Iceberg REST specification

2024-05-29 Thread Jack Ye
to do that based on security concerns. Best, Jack Ye On Wed, May 29, 2024 at 10:28 AM Steven Wu wrote: > Wondering if the auth endpoints can be separated out to a separate OpenAPI > spec file. Then we still have some reference for interactions with auth > server and make it clear it is

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

2024-05-28 Thread Jack Ye
lexity involved >>> will be similar to managing views. >>> >>> Thanks, Ryan, Robert, and Jack, for your input. >>> We will work on publishing the draft spec (inspired by the view spec) >>> this week to facilitate further discussions. >>> >>&g

Re: Addressing security questions in the Iceberg REST specification

2024-05-28 Thread Jack Ye
ther cloud providers next to AWS. > > Kind regards, > Fokko > > > > Op do 23 mei 2024 om 15:49 schreef Dmitri Bourlatchkov > : > >> I think Jack makes a good point with AWS SigV4 Authentication. I suppose, >> in REST Catalog implementations that support tha

Re: Addressing security questions in the Iceberg REST specification

2024-05-28 Thread Jack Ye
g implementations that support that auth method, the > /v1/oauth/token Catalog REST endpoint is redundant. > > Cheers, > Dmitri. > > On Thu, May 23, 2024 at 9:20 AM Jack Ye wrote: > >> I do not know enough details about OAuth to make comments about this >> issue, bu

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

2024-05-28 Thread Jack Ye
s more when we have the actual proposal published. Best, Jack Ye On Tue, May 28, 2024 at 1:32 AM Robert Stupp wrote: > UDFs are as engine specific and portable and "non-centralized" as views > are. The same performance concerns apply to views as well. > Iceberg should define

Re: Addressing security questions in the Iceberg REST specification

2024-05-23 Thread Jack Ye
erg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L72>. It would be nice if we formalize that in the spec, at least define it as a generic authorization header. Best, Jack Ye On Thu, May 23, 2024 at 2:51 AM Robert Stupp wrote: > Hi all, > > Iceberg REST implement

  1   2   3   4   >