Re: Lakehouse Formats with IO/Integration --> Hudi? Iceberg?

2023-11-07 Thread Ismaël Mejía
For iceberg there has been a long time opened issue and some WIP for a sink https://github.com/apache/beam/issues/20327 On Tue, Nov 7, 2023 at 2:08 AM Austin Bennett wrote: > Beam Devs, > > I was looking through GH Issue and online more generally and hadn't seen > much... Has anyone written a

Beam High Priority Issue Report (47)

2023-11-07 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/29214 [Failing Test]: Pyt

Re: Disabling Jenkins Jobs

2023-11-07 Thread Alexey Romanenko
Not sure if it’s related but I see that, seems, all Jenkins jobs are stuck and there is a big Build Queue on https://ci-beam.apache.org/ Random clicks on jobs show that “"All nodes of label ‘beam’ are offline” message. Is it known problem? — Alexey > On 24 Oct 2023, at 21:50, Yi Hu via dev w

Re: Disabling Jenkins Jobs

2023-11-07 Thread Danny McCormick via dev
I don't think it's related. I noticed the problem half an hour ago; it seems there's an expired cert on the Jenkins machines. I'm hoping https://github.com/apache/beam/actions/runs/6786537134/job/18447281366 will fix this since the IO-Datastores cert is the problematic piece I think (and that has f

Re: Disabling Jenkins Jobs

2023-11-07 Thread Yi Hu via dev
Hi Alexey, > all Jenkins jobs are stuck and there is a big Build Queue on https://ci-beam.apache.org/ This is not intentional. This is likely due to INFRA's routine Jenkins upgrade on Nov 5 and caused this outage. Have created https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25147?filte

Re: Disabling Jenkins Jobs

2023-11-07 Thread Alexey Romanenko
Danny, Yi, Thank you for taking care of this! — Alexey > On 7 Nov 2023, at 17:10, Yi Hu via dev wrote: > > Hi Alexey, > > > all Jenkins jobs are stuck and there is a big Build Queue on > > https://ci-beam.apache.org/ > > This is not intentional. This is likely due to INFRA's routine Jenkins

[Question] Error handling for IO Write Functions

2023-11-07 Thread Ramya Prasad via dev
Hello, I am a developer using Apache Beam in my Java application, and I need some help on how to handle exceptions when writing a file to S3. I have tried wrapping my code within a try-catch block, but no exception is being thrown within the try block. I'm assuming that FileIO doesn't throw any ex

Re: [Question] Error handling for IO Write Functions

2023-11-07 Thread Robert Bradshaw via dev
File write failures should be throwing exceptions that will terminate the pipeline on failure. (Generally a distributed runner will make multiple attempts before abandoning the entire pipeline of course.) Are you seeing files failing to be written but no exceptions being thrown? If so, this is def

[VOTE] Release 2.52.0, release candidate #2

2023-11-07 Thread Danny McCormick via dev
Hi everyone, Please review and vote on the release candidate #2 for the version 2.52.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no