Re: [DISCUSS] Dropping Spark 2.4 support

Hi Fokko, Ryan, Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are actively migrating to Spark-3.x and Iceberg 1.1 (or later). I do not anticipate us using Spark-2.4.4 with newer versions of Iceberg (>0.9). If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that should be ok. @Jo

Re: Approaching Vectorized Reading in Iceberg ..

Hi Gautam, You wrote: ' - The filters are not being applied in columnar fashion they are being applied row by row as in Iceberg each filter visitor is stateless and applied separately on each row's column. ' .. this should not be a problem for this particular benchmark as IcebergSourceFlatParquetD

Re: Encouraging performance results for Vectorized Iceberg code

se emails? > There are other people in the community that are interested, like Palantir. > If there isn't anything sensitive then let's try to be more inclusive. > Thanks! > > rb > > On Wed, Aug 7, 2019 at 10:34 PM Anjali Norwood > wrote: > >> Hi Gautam, P

Re: Encouraging performance results for Vectorized Iceberg code

; > On Aug 8, 2019, at 11:56 AM, Anjali Norwood wrote: > > Good suggestion Ryan. Added dev@iceberg now. > > Dev: Please see early vectorized Iceberg performance results a couple > emails down. This WIP. > > thanks, > Anjali. > > On Thu, Aug 8, 2019 at 10:39 AM Ryan

Re: Encouraging performance results for Vectorized Iceberg code(Internet mail)

? What do we learn from >> compare to IcebergSourceFlatParquetDataReadBenchmark.readIceberg? Or >> rather, what numbers are we comparing to? >> >> >> >> -Li >> >> >> >> *发件人**: *Anjali Norwood >> *答复**: *"dev@iceberg.apac

Re: Encouraging performance results for Vectorized Iceberg code(Internet mail)

It is merged now to 'vectorized-read' branch now. Thanks Ryan. -Anjali. On Mon, Aug 12, 2019 at 6:12 PM Anjali Norwood wrote: > Hi Padma, Gautam, All, > > Our (Samarth's and mine) wip vectorized code is here: > https://github.com/anjalinorwood/incubator-iceberg/pu

Re: New committer and PPMC member, Anton Okolnychyi

Congratulations Anton!! Regards Anjali On Tue, Sep 3, 2019 at 2:11 AM Anton Okolnychyi wrote: > Thanks everyone! > > I am really excited to see how the community evolves over time and that we > have more and more great companies and folks joining the project. > I hope we will continue to build

Open issues against Vectorized Iceberg Read milestone

d next steps on Vectorized > read path in Iceberg. Here are my notes from the sync. Feel free to reply > with clarifications in case I mis-quoted or missed anything. > > *Attendees*: > > Anjali Norwood > Padma Pennumarthy > Ryan Blue > Samarth Jain > Gautam Kowshik >

Re: [ANNOUNCE] Apache Iceberg release 0.7.0-incubating

Congratulations!! Regards Anjali On Sun, Oct 27, 2019 at 2:56 PM Ryan Blue wrote: > Here's the release announcement that I just sent out. Thanks to everyone > that contributed to this release! > > -- Forwarded message - > From: Ryan Blue > Date: Sun, Oct 27, 2019 at 2:55 PM > S

Re: Welcome new committer and PPMC member Ratandeep Ratti

Congratulations Ratandeep!! regards. Anjali. On Mon, Feb 17, 2020 at 12:19 AM Manish Malhotra < manish.malhotra.w...@gmail.com> wrote: > Congratulations 🎉!! > > On Sun, Feb 16, 2020 at 8:37 PM RD wrote: > >> Thanks everyone! >> >> -Best, >> R. >> >> On Sun, Feb 16, 2020 at 7:39 PM David Christl

Re: Slack invite

I had the same experience. Please let us know here in dev group when this is fixed .. a bunch of us would like to join. thanks, Anjali. On Thu, Jul 8, 2021 at 3:52 PM Puneet Zaroo wrote: > I tried joining with a guest account and got an error message: "The email > address must match one of the

Re: Slack invite

214 >> and then I realized that someone has also opened >> https://issues.apache.org/jira/browse/INFRA-22088 yesterday. I'm >> happy to watch the link and add a reply to this thread when it appears >> to be working again. >> >> On Thu, Jul 8, 2021 at 1:45

Re: View definitions

Hi Ryan, John Zhuge and I are working on a proposal. We hope to send out a draft for discussion soon. Regards Anjali On Tue, Jul 13, 2021 at 2:29 AM Ryan Murray wrote: > Hi All, > > I was curious if anyone was working on bringing a proposal for Views to > the community? I know extensive work a

Proposal: Support for views in Iceberg

Hello, John Zhuge and I would like to propose the following spec for storing view metadata in Iceberg. The proposal has been implemented [1] and is in production at Netflix for over 15 months. https://docs.google.com/document/d/1wQt57EWylluNFdnxVxaCkCSvnWlI8vVtwfnPjQ6Y7aw/edit?usp=sharing [1] ht

Re: Proposal: Support for views in Iceberg

Murray wrote: > >> Thanks Anjali! >> >> I have left some comments on the document. I unfortunately have to miss >> the community meetup tomorrow but would love to chat more/help w/ >> implementation. >> >> Best, >> Ryan >> >> On

Re: Proposal: Support for views in Iceberg

ations, constructs >>>> and functions) that you found sufficient for expressing your views? >>>> Being able to enumerate them could help validate various approaches >>>> considered here, including feasibility of dedicated representation. >>

Re: Iceberg disaster recovery and relative path sync-up

Thanks for the summary Yufei. Sorry, if this was already discussed, I missed the meeting yesterday. Is there anything in the design that would prevent multiple roots from being in different aws regions? For disaster recovery in the case of an entire aws region down or slow, is metastore still a poi

Re: Iceberg disaster recovery and relative path sync-up

Perfect, thank you Yufei. Regards Anjali On Thu, Aug 12, 2021 at 9:58 PM Yufei Gu wrote: > Hi Anjali, > > Inline... > On Thu, Aug 12, 2021 at 5:31 PM Anjali Norwood > wrote: > >> Thanks for the summary Yufei. >> Sorry, if this was already discussed, I missed the

Re: Identify watermark in the iceberg table properties

+ Sundaram, as he may have some input. regards, Anjali. On Fri, Aug 13, 2021 at 2:41 AM 1 wrote: > Hi,all: > > I need to embed the iceberg table, which is regarded as real-time > table, into our workflow. That is to say, Flink writes data into Iceberg > table in real-time, I need something to

Re: Iceberg disaster recovery and relative path sync-up

table rename happens then we do not move old data / metadata > files, we just change the root location of the new data / metadata files. > If I am correct about this then we might need to handle this differently > for tables with relative paths. > > Thanks, Peter > > On Fri, 1

Re: Iceberg disaster recovery and relative path sync-up

re a few uses for it. >>>> >>>> The more I think about this, the more I like the solution to add >>>> multiple table roots to metadata, rather than removing table roots. Adding >>>> a way to plug in a root selector makes a lot of sense to me and it ensu

Re: Proposal: Support for views in Iceberg

ble to express column projections, simple filter application, >>> joins, grouping and common aggregate and scalar function in a structured >>> IR, how much effort do you think would be required? >>> We didn't really talk about downsides of a structured approach, other &

Re: Proposal: Support for views in Iceberg

Hi All, Please see the spec in markdown format at the PR here to facilitate adding/responding to comments. Please review. thanks, Anjali On Tue, Sep 7, 2021 at 9:31 PM Jack Ye wrote: > Hi everyone, > > I have been thinking about the view support du