Re: How to configure external service for Kafka IO to run the flink job in k8s

Adding a few relevant folks who could help answer this question: @John Casey @Chamikara Jayalath @Robert Bradshaw Lydian, if you have any other information please share an update. Ahmet On Tue, Jun 7, 2022 at 12:49 AM Lydian wrote: > Hi Folks, > > I am trying to set up the Beam environment

Re: [question] Good Course to learn beam

Some other resources: https://beamcollege.dev/ https://www.coursera.org/specializations/serverless-data-processing-with-dataflow#courses - GCP oriented, but primarily focused on Beam. On Tue, Aug 30, 2022 at 10:09 PM P Singh wrote: > Hi, > > Yeah I agree this book covers pretty much what you ne

Re: SSL issue: Kafka Avro write with Schema Registry (GCP)

(moving this to the user list, dev list to bcc.) Adding relevant people: @John Casey . (Keshav, for Dataflow issues you could also reach out to Dataflow support: https://cloud.google.com/dataflow/docs/support) On Mon, Oct 31, 2022 at 1:23 PM Chennakeshavlu Maddela < chennakeshavlu.madd...@davita

Re: Scio 0.12.0 released

Congratulations! On Fri, Oct 28, 2022 at 11:11 AM Claire McGinty wrote: > Hi Scio users! > > We're happy to announce that Scio *0.12.0* > has been released. > This is the culmination of months of work and we're grateful to everyone > who con

Re: Timeline of support for Python 3.10?

Congratulations and thank you! On Tue, Nov 29, 2022 at 1:36 PM Anand Inguva via user wrote: > Hi, > Circling back on this. > > Python 3.10 is available with Apache Beam 2.43.0[1]. > > [1] https://beam.apache.org/blog/beam-2.43.0/ > > Thanks > > On Thu, Jul 21, 2022 at 5:48 PM Lina Mårtensson wr

Re: KafkaIo Metrics

/cc @John Casey - who might be able to help. On Wed, Jan 18, 2023 at 12:43 PM Lydian wrote: > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and > therefore I cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my > KafkaI

Re: Beam SQL Alias issue while using With Clause

Adding: @Andrew Pilloud @Kenneth Knowles On Thu, Jan 12, 2023 at 12:31 PM Talat Uyarer via user wrote: > Hi All, > > I am using Beam 2.43 with Calcite SQL with Java. > > I have a query with a WITH clause and some aliasing. Looks like Beam Query > optimizer after optimizing my query, it drops S

Re: [Question] 2.47.0 Release

Hi! On Sat, Mar 18, 2023 at 11:53 AM Ayoyinka Obisesan < ayoyinkaobise...@gmail.com> wrote: > cc-ing: user@beam.apache.org > > Please see the questions in the above email. > > Kind regards, > Ayoyinka. > > > On Fri, Mar 17, 2023 at 11:40 AM Ayoyinka Obisesan < > ayoyinkaobise...@gmail.com> wrote:

Re: Beam Playground (Beta) - Interactive Learning for Apache Beam

This is such an amazing improvement. Thank you all for making this happen! On Wed, May 24, 2023 at 12:41 PM Alex Kosolapov wrote: > Hi Beam community! > > > > We are inviting you to try out the new Apache Beam Playground > [1] version - an interactive environment

Re: [Launch Announcement] Beam Quest

Thank you Svetak! I would encourage everyone to try out, and get the badges :) On Fri, Jun 9, 2023 at 7:03 AM Svetak Sundhar via user wrote: > Hi Beam Community, > > We're excited to launch the "Getting Started with Apache Beam" Quest > . This ques

Re: [Invite] Slack #beam channel request

I sent an invite to you. On Wed, Jul 26, 2023 at 3:25 PM Sheldon Teerlink wrote: > Can I please get an invite to the #beam channel within the Slack ASF > workspace? > > Regards, > Sheldon >

RE: Missing Beam Katas in Intellij >=2023.3

Hi Bartosz, Thanks for flagging this. Adding @Austin Bennett and @Israel Herraiz -- They were the two people who maintained beam katas or helped with related questions before. Ahmet On 2023/07/27 10:21:24 Bartosz Zabłocki via user wrote: > Hi all, > I'd like to bring to your attention that Be

Re: [question][python][flink][kafka] Configuring java worker

Adding relevant folks who might be able to help: @Chamikara Jayalath @John Casey On Wed, Aug 2, 2023 at 4:57 PM marc hurabielle wrote: > Hello beam community, > > I have a question arround beam python, and kafka and flink runner. > As the kafka source need java, right now, the flink runner wil

Re: Is Pulsar IO Connector Officially Supported?

Pulsar IO did not change much since it was originally added in 2022. You can find about the gaps in this presentation ( https://2022.beamsummit.org/slides/Developing%20PulsarIO%20Connector.pdf) starting slide 52 (next steps). That might give you the background information to make an informed decisi

Re: Apache Bean on GCP / Forcing to use py 3.11

If you could use py 3.11 locally, you will get python 3.11 in your cloud environment as well. Is that not happening? When you run Apache Beam on GCP, the python version you are using in your local virtual environment will be used in the cloud environment as well. I believe this is true for non-GCP

Re: Question on slowly updating global window side inputs

Thank you for the follow up. If you think that presentation is useful, and this is not properly captured in docs, would you be kind enough to help us improve our docs? :) It could be a link to that deck, and github issue, or new content in docs based on that presentation. On Sat, Aug 3, 2024 at

Re: Question on slowly updating global window side inputs

, Jaehyeon Kim wrote: > >> Thank you for the suggestion. Let me think about how to contribute and >> take an action. >> >> On Tue, 13 Aug 2024, 8:50 am Ahmet Altay via user, >> wrote: >> >>> Thank you for the follow up. >>> >>> If you

Re:

Hi Ahijah -- Email did not have your question. Do you have a question? (moving dev list to bcc.) On Mon, Sep 16, 2024 at 12:57 PM Ahijah Koil Boaz Isacejayakumar via dev < d...@beam.apache.org> wrote: > > > This message contains proprietary information from Equifax which may be > confidential. I

Is it possible to set a compression coder for the golang pardo?

(Asking a question on a user's behalf, question is in the title.) Ahmet

Re: [Question] Regarding custom metrics in beam

Adding Jan & Ken as people with potential knowledge in this area. On Sun, Oct 20, 2024 at 10:43 PM Aditya Shirsath wrote: > Hello, > > I'm creating a beam pipeline to run on flink runner. I've created > custom metrics in beam(JAVA SDK). But I'm unable to see it in flink > dashboard or flink rest

Re: Apache beam error java

Hi Zack, Have you seen Tomo's response (below)? Were you able to resolve your issue? Ahmet On Tue, Oct 29, 2024 at 5:46 PM Tomo Suzuki via user wrote: > I think you have incompatible library versions in your dependencies. Would > you share your dependency graph? > > On Tue, Oct 29, 2024 at 8:0

Re: JMSIO AutoScaler

Adding Zack explicitly. On Fri, Nov 15, 2024 at 2:57 AM Radek Stankiewicz via user < user@beam.apache.org> wrote: > hey Zach, > > Few recommendations: > - level of parallelism dataflow v1 is set > by desiredNumUnboundedSourceSplits (default N=0) which means that JmsIO > will create N splits equal

Re: Beam 2.61.0 Release

Thank you all, congratulations! On Mon, Nov 25, 2024 at 1:03 PM XQ Hu via user wrote: > Great job! Thanks for your work! > > On Mon, Nov 25, 2024 at 3:45 PM Danny McCormick via user < > user@beam.apache.org> wrote: > >> Hi, I am happy to announce that Beam 2.61.0 has been fully released. For >>

Re: Number of connections to Kafka

It should help. Adding @Yi Hu @Steven van Rossum @Sam Whittle who would be able to give a more definitive answer. On Fri, Feb 7, 2025 at 9:07 AM Utkarsh Parekh wrote: > Hi Team, > > > > I came across this PR and wanted to check if it addresses the issue of > multiple kafka connections being c

Re: [Question] Timeout expired while fetching topic metadata

(explicitly adding Utkarsh). On Wed, Nov 20, 2024 at 10:13 AM John Casey via user wrote: > Hi Utkarsh, > > I can't say for certain without more details, but timeouts against Kafka > usually represent auth issues. > > By and large, when a user tries to connect but doesn't have permission, > Kafka

Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0

I do not know the answer. Adding @Kenneth Knowles and @XQ Hu in case they might be able to answer it. On Sun, Jan 26, 2025 at 3:54 PM Lydian Lee wrote: > Hi, > > We are trying to bump an old pipeline using flink runner and beam python > SDK. > The version changes are: > - Flink: 1.15.4 -> 1.18

Re: JMSIO support

I do not know the answer. Adding @Radek Stankiewicz & @Yi Hu in case they might be able to help. On Mon, Jan 27, 2025 at 2:55 PM Zack Culberson < zack.culber...@albertsons.com> wrote: > Hi all, > > I was wondering if anyone would know, we are using the JMS IO read to read > from 3 IBM MQ’s. We

Re: Beam YAML is great!

Great to hear and thank you for the feedback Joey! Would you be interested in publishing a case study on Beam's website? We will all very much appreciate that :) On Tue, Apr 29, 2025 at 2:41 PM Joey Tran wrote: > We've just upgraded beam to 2.63 and started prototyping and building on > Beam YA

Re: [python] Beam Education Material for Workshops

Adding to XQ's list, there are also some docs discussing the execution model in addition to the programming model - https://cloud.google.com/dataflow/docs/pipeline-lifecycle - https://cloud.google.com/dataflow/docs/concepts/beam-programming-model & https://beam.apache.org/documentation/programming-