Re: [ANNOUNCE] Spark portable runner (batch) now available for Java, Python, Go

Thank you Kyle! This is great news :) On Mon, Jun 17, 2019 at 6:40 AM Andres Angel wrote: > Really great achievement!!! congrats. > > On Mon, Jun 17, 2019 at 7:49 AM Robert Bradshaw > wrote: > >> Excellent work, very excited to see this! >> >> On Fri, Jun 14, 2019 at 11:02 PM Kyle Weaver wrote

Re: [python SDK] Returning Pub/Sub message_id and timestamp

When with_attributes is set to True, the elements will be of type PubsubMessage [1]. I could not find a test/example for this, but documentation suggests [2], PubsubMessage will have an attributes map including the system provided values. One of those keys will be message_id [3]. [1] https://githu

Re: [python SDK] Returning Pub/Sub message_id and timestamp

> > On Wed, 2019-07-24 at 09:16 -0700, Ahmet Altay wrote: > > *This message originated from outside your organization* > -- > When with_attributes is set to True, the elements will be of type > PubsubMessage [1]. I could not find a test/

Re: [python SDK] Returning Pub/Sub message_id and timestamp

is not great, I'm assuming the latter option would need to look > something like this? > > attributes = dict((key, msg.attributes[key]) for key in msg.attributes) > attributes.update({'message_id': msg.message_id, 'publish_time':

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

Is not this flag set automatically for the portable runner here [1] ? [1] https://github.com/apache/beam/blob/f0aa877b8703eed4143957b4cd212aa026238a6e/sdks/python/apache_beam/pipeline.py#L160 On Tue, Sep 17, 2019 at 2:07 PM Robert Bradshaw wrote: > On Tue, Sep 17, 2019 at 1:43 PM Thomas Weise

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

stance of PortableRunner. > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > On Tue, Sep 17, 2019 at 2:14 PM Ahmet Altay wrote: > >> Is not this flag set automatically for the portable runner here [1] ? >> >&g

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

he root > of the problem. > I might be wrong, IIUC, it will temporarily resolve the reported issues. Is this not accurate? > > -Max > > On 17.09.19 14:20, Ahmet Altay wrote: > > Could you make that change and see if it would have addressed the issue > > here? > >

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

aver | Software Engineer | github.com/ibzib > > <http://github.com/ibzib> | kcwea...@google.com kcwea...@google.com> > > > > > > On Tue, Sep 17, 2019 at 3:38 PM Ahmet Altay > <mailto:al...@google.com>> wrote: > > > > > > > >

Re: [ANNOUNCE] Beam 2.18.0 Released

Thank you Udi! On Tue, Jan 28, 2020 at 2:13 PM kant kodali wrote: > Looks like > https://beam.apache.org/documentation/runners/capability-matrix/ needs to > be updated? since there seems to be support for spark structured streaming? > > On Tue, Jan 28, 2020 at 1:47 PM Connell O'Callaghan > wrot

Re: Bay Area Beam Meetup 19 Feb (Last Wednesday).

Thank you for sharing talks! On Fri, Feb 21, 2020 at 9:18 PM Austin Bennett wrote: > Hi All, > > We had a meetup @Sentry.io on Wednesday -- with a solid 40+ engaged > attendees. > > Thanks for those that joined in person, and for those that were unable, > talks can be found online --> > Syd's ta

Re: [ANNOUNCEMENT] Support for Hadoop 3 confirmed

Awesome! Thank you Piotr! On Tue, Nov 24, 2020 at 4:45 AM Piotr Szuberski wrote: > > PS: What’s about Hive/HBase modules? > > > > HCatalog doesn't want the tests to be passed with Hadoop3 - I tried to > figure out why but I gave up. > > HBase - there is an effort to test Beam against HBase 1 and

Re: Help measuring upcoming performance increase in flink runner on production systems

Hi Teodor, Thank you for working on this. If I remember correctly, there were some opportunities to improve in the previous paper (e.g. not focusing deprecated runners, long running benchmarks, varying data sizes). And I am excited that you are keeping the community as part of your research proces

Re: [VOTE] Release 2.27.0, release candidate #1

+1 (binding). I validated python quickstarts. Thank you Pablo. On Tue, Dec 22, 2020 at 10:04 PM Jean-Baptiste Onofre wrote: > +1 (binding) > > Regards > JB > > Le 23 déc. 2020 à 06:46, Pablo Estrada a écrit : > > Hi everyone, > Please review and vote on the release candidate #1 for the version

Re: [VOTE] Release 2.27.0, release candidate #4

+1 (binding) - validated python quickstarts. Thank you Pablo. On Wed, Jan 6, 2021 at 1:57 PM Pablo Estrada wrote: > +1 (binding) > I've built and unit tested existing Dataflow Templates with the new > version. > Best > -P. > > On Tue, Jan 5, 2021 at 11:17 PM Pablo Estrada wrote: > >> Hi everyo

Re: Python SDK and Kerberos

/cc +Udi Meiri On Wed, Feb 3, 2021 at 9:23 AM Doutre, Mark wrote: > Hi, > I’m trying to use the python sdk to write data to hdfs. However our > cluster is kerberized. Is it possible to do this with the current sdk? If > it is, how to you get it to authenticate? > > Everything works fine fro

Re: getting windowparam in python

/cc +Yichi Zhang On Fri, Feb 19, 2021 at 2:24 AM Manninger, Matyas < matyas.mannin...@veolia.com> wrote: > Dear Beam users, > > I am using the following code to log debug info about my streaming > pipeline: > > class DebugWindowInformation(beam.DoFn): > def to_runner_api_parameter(self, unus

Re: Problems with conflicting option names when running tests on 2.28.0

/cc +Valentyn Tymofieiev - might have an idea. On Sat, Feb 27, 2021 at 2:42 AM adam moore wrote: > Hi there beam-team. > > I've recently upgraded to 2.28.0 but I'm having problems running tests > which were previously working on 2.27.0. > > Here's a description of the situation, and I'm wonderi

Re: [Meetup] Wednesday 17th March 9am PDT / 5pm CET - Emily Ye on Custom Containers

This is great and could we share this on our Twitter account? It will reach more users that way. /cc @Brittany Hermann @Pablo Estrada On Mon, Mar 15, 2021 at 7:32 AM Max King wrote: > Hi Everyone, > > On Wednesday there will be an online Beam meetup with Emily Ye. Join > Emily at 9am PDT / 5p

Re: Apache Beam Python SDK ReadFromKafka does not receive data

On Wed, Mar 17, 2021 at 10:23 AM Chamikara Jayalath wrote: > IIUC, currently Splittable DoFn (source framework) does not work for > portable runners in streaming mode due to the issue Boyuan mentioned. > > On Tue, Mar 16, 2021 at 8:35 PM Sumeet Malhotra > wrote: > >> Thanks Cham. In the python

Re: Source advancing before previously read records are processed fully

/cc @Boyuan Zhang On Wed, Mar 17, 2021 at 3:38 AM Pradyumna Achar wrote: > Hello, > > I am running into a strange issue with the KafkaIO streaming source. > > The source just keeps reading records from the Kafka topics even before > the downstream DoFns in the pipeline have got a chance to proc

Re: Scio 0.10.1 released

Congratulations! On Thu, Apr 1, 2021 at 1:44 PM Kirill Panarin wrote: > Hi all, > > We just released Scio 0.10.1 *"Belli Horizontis"*. Here's a short summary > of the notable changes since 0.10.0: > > 🚀 Enhancements > >- Support verifyDeterministic and consistentWithEquals in recursive >

Re: Inaccessible CI-built python SDK artifact on GCS staging bucket (individual wheels are accessible)

Moving to dev list. This looks like a bug in the build workflow but I cannot pinpoint it. In the GCS UI all objects have Public Access: Not Authorized property. This explains why you cannot download them. However workflow uses "gsutil cp -r -a public-read ..." to copy the object to GCS, and "-a pu

Re: Issues running Kafka streaming pipeline in Python

/cc @Boyuan Zhang for kafka @Chamikara Jayalath for multi language might be able to help. On Tue, Jun 1, 2021 at 9:39 PM Alex Koay wrote: > Hi all, > > I have created a simple snippet as such: > > import apache_beam as beam > from apache_beam.io.kafka import ReadFromKafka > from apache_beam.op

Apache Beam, version 0.6.0 with Python SDK

the recommended version; we encourage everyone to upgrade from any earlier releases. We thank all users and contributors who have helped make this release possible. If you haven't already, we'd like to invite you to join us, as we work towards our first release with API stability. - A

Re: Best way to load heavy object into memory on nodes (python sdk)

You can see an example implementation of Luke's suggestion in the tensorflow-transform project [1]. Thread local is used in that case, this will work for runners that re-use the same thread to execute bundles. [1] *https://github.com/tensorflow/transform/blob/master/tensorflow_transform/beam/impl

Re: Best way to load heavy object into memory on nodes (python sdk)

If you ever stop by > Stockholm I'll be happy to buy you guys a beer! > > > On Wed, May 24, 2017 at 6:38 PM, Ahmet Altay wrote: > >> You can see an example implementation of Luke's suggestion in the >> tensorflow-transform project [1]. Thread local is used in tha

Re: Practices for running Python projects on Dataflow

Sébastien, what kind of an issue you had with using setup.py with installation_requires? On Mon, Jun 5, 2017 at 4:44 PM, Morand, Sebastien < sebastien.mor...@veolia.com> wrote: > Hi, > > I ran into trouble when using setup.py with installation_requires. So I > basically ended up with setup.py wit

Re: Error in Python SDK 2.0 dataflow

Hi Sébastien, Could you explain more on what you are doing? Is it possible that you are overwriting/removing google-apitools package somehow? Also, for dataflow service issues feel free to use one of the methods mentioned in Dataflow support page [1]. Thank you, Ahmet [1] https://cloud.google.co

Re: Error in Python SDK 2.0 dataflow

The original error reported is unlikely to be related to C-dependent library. Please let us know if you get more information. On Mon, Jun 5, 2017 at 5:09 PM, Dmitry Demeshchuk wrote: > Interesting, I'm running into a very similar issue. My current wild > guesses are: > > 1. I'm somehow messing u

Re: Installing non-native Python dependencies in Dataflow

Hi, Please see Managing Python Pipeline Dependencies [1] for various ways on installing additional dependencies. The section on non-python dependencies is relevant to your question. Thank you, Ahmet [1] https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ On Mon, Jun 5, 2017

Re: Installing non-native Python dependencies in Dataflow

#x27;m doing wrong (Dataflow doesn't send out installation failure logs to > Stackdriver, only runtime logs, so it seems). > Have you tried looking worker-startup logs? All of the logs should be in stackdriver. > > On Tue, Jun 6, 2017 at 9:21 AM, Ahmet Altay wrote: > >>

Re: Installing non-native Python dependencies in Dataflow

the new command has been generated with a newer version of setuptools > (specifically, my project has setuptools==36.0.1 as a dependency of some > package). I'm still digging more through the Stackdriver logs but so far > couldn't find out the exact reason of the fai

Re: Installing non-native Python dependencies in Dataflow

dependency >> packages that depends on that specific version. >> >> Thanks for the information, I'll try to explicitly install 33.1.1 and see >> if it changes anything. >> >> On Tue, Jun 6, 2017 at 7:13 PM, Ahmet Altay wrote: >> >>> Pinning s

Re: Order of module loading in the Dataflow Python runner

Dmitry, You can use the --beam_plugin flag [1] to import your plugin before any other code execution. A failure in this step will result in the warning Robert mentioned ("Failed to import beam plugin ..."), and you can look for those in worker logs. This flag is not available in the 2.0.0 version

Re: Order of module loading in the Dataflow Python runner

ins though. Is > that just an arbitrary Python package, or is it supposed to be an addition > to the apache_beam package, akin to "apache_beam[gcp]" ? > Arbitrary package. It is a mechanism to import modules before executing things. This flag is likely the existing solution for BEAM-2573

Re: [Python] Stateful processing in Python SDK

HI Vilhelm, Python SDK currently does not support stateful processing. We should update the capability matrix to show this. I filed https://issues.apache.org/jira/browse/BEAM-2687 to track this feature. Feel free to follow it there or better make it happen. As far as I know, nobody is actively wor

Re: Provide docker image

Hi Sebastien, Python Dataflow jobs typically have a ~2 minute startup time per VM, About 1 minute of that is spinning up the VM and another minute is installing user code and dependencies. By using a custom container (when that is available) you might shave some time off from the latter, but you w

Apache Beam, version 2.1.0

, contribution and participation through our mailing lists, issue tracker, pull requests, and events. - Ahmet Altay, on behalf of the Apache Beam community. [1] https://beam.apache.org/get-started/downloads/ [2] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12340528

Re: Reading message attributes in PubSub source in Python

On Tue, Jan 23, 2018 at 2:48 PM, Dmitry Demeshchuk wrote: > Hi list, > > My understanding is that ReadStringsFromPubSub > > doesn't > provide any way of getting the message metadata (attributes, publish > t

Re: dataflow HDF5 loading pipeline errors

Hi Eila, The error "work item was attempted 4 times without success" indicates that some operation is consistently failing. You can find more information in Dataflow worker logs [1] about the actual error. I cannot tell for sure without looking at the logs, I suspect your issue is related to proc

Re: Apache beam DataFlow runner throwing setup error

Hi Rajesh, Have you looked at the worker-startup logs [1]? You should be able to see the setup error there. It is possible that something in your requirements file is failing to install in the workers. If that is the case, see Managing Python Pipeline Dependencies [2] for alternative options. You

Re: How to decide how much quota do I need

+ user, dev to bcc Eila, there is some information here: https://cloud.google.com/dataflow/quotas on quotas in general. Specifically for in use IP addresses, you can look at autoscaling messages and see what was autoscaling trying to upscale to. It is also possible to use large machine types (e.g

Re: executing the pipeline from datalab

+ user, dev to bcc Eila, Is it possible that you are using an old version? I remember pending was missing in the dictionary and was added later. If that is not the reason, could you file a JIRA issue? Thank you, Ahmet On Fri, Mar 23, 2018 at 6:15 AM, Jean-Baptiste Onofré wrote: > Hi Eila, >

Re: Dataflow throwing backend error

Hi Rajesh, This looks like a transient error from GCS. Beam SDK will retry tasks in the face of such errors and those typically do not make your pipeline fail. If you have additional questions please reach out to Dataflow support ( https://cloud.google.com/dataflow/support). Thank you, Ahmet On

Re: Pipeline is passing on local runner and failing on Dataflow runner - help with error

Hi Ella, It seems like, the package related to indexes.base is not installed in the workers. Could you try one of the methods in "Managing Python Pipeline Dependencies" [1], to stage that dependency? Ahmet [1] https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ On Thu, Jun

Re: Pipeline is passing on local runner and failing on Dataflow runner - help with error

gt; > > On Thu, Jun 21, 2018 at 1:55 PM, Ahmet Altay wrote: > >> Hi Ella, >> >> It seems like, the package related to indexes.base is not installed in >> the workers. Could you try one of the methods in "Managing Python Pipeline >> Dependencies" [1], to

Re: Using user developped source in streamline python

Hi Sébastien, Currently there is no work in progress for including the write transforms for the locations you listed. You could develop your own version if interested. Please see WriteToBigquery transform [1] for reference. Ahmet [1] https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dc

Re: google.cloud.bigQuery version on workers - please HELP

Hi Eila, You can find a list of dependencies installed in Dataflow workers in [1]. Dataflow workers will have a set of dependencies that will satisfy the requirements from setup.py. Which bigquery library you are using? There is a google-cloud-bigquery==0.25.0 dependency, I am not sure where the

Re: google.cloud.bigQuery version on workers - please HELP

issues due to the API differences. (Details: https://github.com/apache/beam/pull/5895). There is a recent push for Beam to upgrade all dependencies to their latest version, and I I assume this will be addressed as part of it. Unfortunately, before that fix it is not possible to use the latest vers

Re: How to inforce google-cloud-bigquery==0.28.0 installation on the dataflow workers

I answered your similar question in the other thread [1]. Let me know if you have a separate question. Ahmet [1] https://lists.apache.org/thread.html/0d1e0ff046657a5439c834e23da375dc405cb562017b08a508b6e079@%3Cuser.beam.apache.org%3E On Fri, Jul 13, 2018 at 11:04 AM, OrielResearch Eila Arich-Lan

Re: Scio 0.5.6 released

Congratulations! Rooting for pica pica to be the code name for the next release :) On Wed, Jul 25, 2018 at 4:57 PM, Neville Li wrote: > Hi all, > > We just released Scio 0.5.6. This release includes a lot of improvements, > bug fixes, and is based on the latest Beam 2.5.0. > > https://github.co

[Feedback Request] Long term support releases

Hi all, In order to increase the predictability of Beam releases, I proposed introducing long term support releases with 12 month defined support periods [1]. I would like open this discussion to our user@ community and receive your feedback on it. We will appreciate any input. Here is a draft PR

Re: INFO:root:Executing Error when executing a pipeline on dataflow

Ella, I believe you are using version 2.0.0. Use of --download flag is fixed at head. (I do not recall the exact version of the fix, could be 2.2 or 2.3). If possible please try to use a newer version of Beam. Also, as Luke suggested, we would welcome any contributions to the documentation. On Fr

[ANNOUNCE] Apache Beam 2.8.0 released!

ails: https://beam.apache. org/blog/2018/10/29/beam-2.8.0.html Thanks to everyone who contributed to this release, and we hope you enjoy using Beam 2.8.0. -- Ahmet Altay, on behalf of The Apache Beam team

[ANNOUNCE] Apache Beam 2.11.0 released!

. -- Ahmet Altay, on behalf of The Apache Beam team

Re: PipelineOptions at execution time from DirectRunner

On Thu, Mar 21, 2019 at 4:20 PM Pablo Estrada wrote: > Hi all, > The DirectRunner does not seem to support RuntimeValueProvider. Is there a > suggestion for DirectRunner pipelines to access arguments passed in as > pipeline options(but not necessarily passed explicitly by users) at > pipeline exe

Re: PipelineOptions at execution time from DirectRunner

On Thu, Mar 21, 2019 at 5:29 PM Pablo Estrada wrote: > Thanks Ahmet! These are illustrative explanations. > > I still wonder about one question: > >> >>> Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is >>> a possiblity, but it seems like that's not ideal. Any other suggest

Re: Implementation an S3 file system for python SDK

+Pablo Estrada On Wed, Apr 3, 2019 at 8:46 AM Lukasz Cwik wrote: > +dev > > On Wed, Apr 3, 2019 at 2:03 AM Pasan Kamburugamuwa < > pasankamburugamu...@gmail.com> wrote: > >> Hi , >> >> I have completed a proposal to implementation an S3 file system for >> python SDK for the google summer of Co

Re: Implementation an S3 file system for python SDK - Updated

+dev +Pablo Estrada +Chamikara Jayalath +Udi Meiri Thank you Pasan. I quickly looked at the proposal and it looks good. Added a few folks who could offer additional feedback. On Mon, Apr 8, 2019 at 12:13 AM Pasan Kamburugamuwa < pasankamburugamu...@gmail.com> wrote: > Hi, > > I have updated

Re: Debugging Python Dataflow workflow installation failure

Hi Chris, I am guess this is the same question as the StackOverflow question ( https://stackoverflow.com/questions/55729163/google-dataflow-with-python-failed-to-install-workflow-exit-status-1). I dropped a comment there for clarification. Let's use StackOverflow for this question. Ahmet On Mon,

Re: How to configure external service for Kafka IO to run the flink job in k8s

Adding a few relevant folks who could help answer this question: @John Casey @Chamikara Jayalath @Robert Bradshaw Lydian, if you have any other information please share an update. Ahmet On Tue, Jun 7, 2022 at 12:49 AM Lydian wrote: > Hi Folks, > > I am trying to set up the Beam environment

Re: [question] Good Course to learn beam

Some other resources: https://beamcollege.dev/ https://www.coursera.org/specializations/serverless-data-processing-with-dataflow#courses - GCP oriented, but primarily focused on Beam. On Tue, Aug 30, 2022 at 10:09 PM P Singh wrote: > Hi, > > Yeah I agree this book covers pretty much what you ne

Re: SSL issue: Kafka Avro write with Schema Registry (GCP)

(moving this to the user list, dev list to bcc.) Adding relevant people: @John Casey . (Keshav, for Dataflow issues you could also reach out to Dataflow support: https://cloud.google.com/dataflow/docs/support) On Mon, Oct 31, 2022 at 1:23 PM Chennakeshavlu Maddela < chennakeshavlu.madd...@davita

Re: Scio 0.12.0 released

Congratulations! On Fri, Oct 28, 2022 at 11:11 AM Claire McGinty wrote: > Hi Scio users! > > We're happy to announce that Scio *0.12.0* > has been released. > This is the culmination of months of work and we're grateful to everyone > who con

Re: Timeline of support for Python 3.10?

Congratulations and thank you! On Tue, Nov 29, 2022 at 1:36 PM Anand Inguva via user wrote: > Hi, > Circling back on this. > > Python 3.10 is available with Apache Beam 2.43.0[1]. > > [1] https://beam.apache.org/blog/beam-2.43.0/ > > Thanks > > On Thu, Jul 21, 2022 at 5:48 PM Lina Mårtensson wr

Re: KafkaIo Metrics

/cc @John Casey - who might be able to help. On Wed, Jan 18, 2023 at 12:43 PM Lydian wrote: > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and > therefore I cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my > KafkaI

Re: Beam SQL Alias issue while using With Clause

Adding: @Andrew Pilloud @Kenneth Knowles On Thu, Jan 12, 2023 at 12:31 PM Talat Uyarer via user wrote: > Hi All, > > I am using Beam 2.43 with Calcite SQL with Java. > > I have a query with a WITH clause and some aliasing. Looks like Beam Query > optimizer after optimizing my query, it drops S

Re: [Question] 2.47.0 Release

Hi! On Sat, Mar 18, 2023 at 11:53 AM Ayoyinka Obisesan < ayoyinkaobise...@gmail.com> wrote: > cc-ing: user@beam.apache.org > > Please see the questions in the above email. > > Kind regards, > Ayoyinka. > > > On Fri, Mar 17, 2023 at 11:40 AM Ayoyinka Obisesan < > ayoyinkaobise...@gmail.com> wrote:

Re: Beam Playground (Beta) - Interactive Learning for Apache Beam

This is such an amazing improvement. Thank you all for making this happen! On Wed, May 24, 2023 at 12:41 PM Alex Kosolapov wrote: > Hi Beam community! > > > > We are inviting you to try out the new Apache Beam Playground > [1] version - an interactive environment

Re: [Launch Announcement] Beam Quest

As this was one of GCP's first OSS quests, there were many people > instrumental in making this possible. > > Thanks to: > -Danielle Syse > -Ajay Hemnani > -Joellen Saunders > -Grzegorz Wierzchows > -Ahmet Altay > -XQ Hu > -Jenny Palomino > -Svetak Sundhar > -Shunping Huang > > Thanks, > > > > Svetak Sundhar > > Data Engineer > s vetaksund...@google.com > >

Re: [Invite] Slack #beam channel request

I sent an invite to you. On Wed, Jul 26, 2023 at 3:25 PM Sheldon Teerlink wrote: > Can I please get an invite to the #beam channel within the Slack ASF > workspace? > > Regards, > Sheldon >

RE: Missing Beam Katas in Intellij >=2023.3

Hi Bartosz, Thanks for flagging this. Adding @Austin Bennett and @Israel Herraiz -- They were the two people who maintained beam katas or helped with related questions before. Ahmet On 2023/07/27 10:21:24 Bartosz Zabłocki via user wrote: > Hi all, > I'd like to bring to your attention that Be

Re: [question][python][flink][kafka] Configuring java worker

Adding relevant folks who might be able to help: @Chamikara Jayalath @John Casey On Wed, Aug 2, 2023 at 4:57 PM marc hurabielle wrote: > Hello beam community, > > I have a question arround beam python, and kafka and flink runner. > As the kafka source need java, right now, the flink runner wil

Re: Is Pulsar IO Connector Officially Supported?

Pulsar IO did not change much since it was originally added in 2022. You can find about the gaps in this presentation ( https://2022.beamsummit.org/slides/Developing%20PulsarIO%20Connector.pdf) starting slide 52 (next steps). That might give you the background information to make an informed decisi

Re: Apache Bean on GCP / Forcing to use py 3.11

If you could use py 3.11 locally, you will get python 3.11 in your cloud environment as well. Is that not happening? When you run Apache Beam on GCP, the python version you are using in your local virtual environment will be used in the cloud environment as well. I believe this is true for non-GCP

Re: Question on slowly updating global window side inputs

Thank you for the follow up. If you think that presentation is useful, and this is not properly captured in docs, would you be kind enough to help us improve our docs? :) It could be a link to that deck, and github issue, or new content in docs based on that presentation. On Sat, Aug 3, 2024 at

Re: Question on slowly updating global window side inputs

, Jaehyeon Kim wrote: > >> Thank you for the suggestion. Let me think about how to contribute and >> take an action. >> >> On Tue, 13 Aug 2024, 8:50 am Ahmet Altay via user, >> wrote: >> >>> Thank you for the follow up. >>> >>> If you

Re:

Hi Ahijah -- Email did not have your question. Do you have a question? (moving dev list to bcc.) On Mon, Sep 16, 2024 at 12:57 PM Ahijah Koil Boaz Isacejayakumar via dev < d...@beam.apache.org> wrote: > > > This message contains proprietary information from Equifax which may be > confidential. I

Is it possible to set a compression coder for the golang pardo?

(Asking a question on a user's behalf, question is in the title.) Ahmet

Re: [Question] Regarding custom metrics in beam

Adding Jan & Ken as people with potential knowledge in this area. On Sun, Oct 20, 2024 at 10:43 PM Aditya Shirsath wrote: > Hello, > > I'm creating a beam pipeline to run on flink runner. I've created > custom metrics in beam(JAVA SDK). But I'm unable to see it in flink > dashboard or flink rest

Re: Apache beam error java

Hi Zack, Have you seen Tomo's response (below)? Were you able to resolve your issue? Ahmet On Tue, Oct 29, 2024 at 5:46 PM Tomo Suzuki via user wrote: > I think you have incompatible library versions in your dependencies. Would > you share your dependency graph? > > On Tue, Oct 29, 2024 at 8:0

Re: JMSIO AutoScaler

Adding Zack explicitly. On Fri, Nov 15, 2024 at 2:57 AM Radek Stankiewicz via user < user@beam.apache.org> wrote: > hey Zach, > > Few recommendations: > - level of parallelism dataflow v1 is set > by desiredNumUnboundedSourceSplits (default N=0) which means that JmsIO > will create N splits equal

Re: Beam 2.61.0 Release

Thank you all, congratulations! On Mon, Nov 25, 2024 at 1:03 PM XQ Hu via user wrote: > Great job! Thanks for your work! > > On Mon, Nov 25, 2024 at 3:45 PM Danny McCormick via user < > user@beam.apache.org> wrote: > >> Hi, I am happy to announce that Beam 2.61.0 has been fully released. For >>

Re: Number of connections to Kafka

It should help. Adding @Yi Hu @Steven van Rossum @Sam Whittle who would be able to give a more definitive answer. On Fri, Feb 7, 2025 at 9:07 AM Utkarsh Parekh wrote: > Hi Team, > > > > I came across this PR and wanted to check if it addresses the issue of > multiple kafka connections being c

Re: [Question] Timeout expired while fetching topic metadata

(explicitly adding Utkarsh). On Wed, Nov 20, 2024 at 10:13 AM John Casey via user wrote: > Hi Utkarsh, > > I can't say for certain without more details, but timeouts against Kafka > usually represent auth issues. > > By and large, when a user tries to connect but doesn't have permission, > Kafka

Re: Unable to run in python-worker-harness after bumpping from 2.41.0 to 2.60.0

I do not know the answer. Adding @Kenneth Knowles and @XQ Hu in case they might be able to answer it. On Sun, Jan 26, 2025 at 3:54 PM Lydian Lee wrote: > Hi, > > We are trying to bump an old pipeline using flink runner and beam python > SDK. > The version changes are: > - Flink: 1.15.4 -> 1.18

Re: JMSIO support

I do not know the answer. Adding @Radek Stankiewicz & @Yi Hu in case they might be able to help. On Mon, Jan 27, 2025 at 2:55 PM Zack Culberson < zack.culber...@albertsons.com> wrote: > Hi all, > > I was wondering if anyone would know, we are using the JMS IO read to read > from 3 IBM MQ’s. We

Re: Beam YAML is great!

Great to hear and thank you for the feedback Joey! Would you be interested in publishing a case study on Beam's website? We will all very much appreciate that :) On Tue, Apr 29, 2025 at 2:41 PM Joey Tran wrote: > We've just upgraded beam to 2.63 and started prototyping and building on > Beam YA

Re: [python] Beam Education Material for Workshops

Adding to XQ's list, there are also some docs discussing the execution model in addition to the programming model - https://cloud.google.com/dataflow/docs/pipeline-lifecycle - https://cloud.google.com/dataflow/docs/concepts/beam-programming-model & https://beam.apache.org/documentation/programming-