Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam

Hello, I’m encountering the following error while working with ReadFromKafka with the Dataflow runner V2 in Apache Beam: 2025-02-25 23:14:19.435 PST Error translating pipeline. Unknown non-composite transform urn beam:transform:read:v1 for external_1Reading-messages-from-Kafka-KafkaIO-Read-Ka

Number of connections to Kafka

Hi Team, I came across this PR and wanted to check if it addresses the issue of multiple kafka connections being created by Dataflow jobs. https://github.com/apache/beam/pull/32928 Would appreciate any insights. Thank you, -- Utkarsh Warning: All e-mail sent t

Commit Kafka message after processing

Hello everyone, For Python SDK users, what configuration do you use in the beam pipeline to ensure committing the offset after loading data in BigQuery? What's the best approach? Utkarsh

[Question] Timeout expired while fetching topic metadata

 Hi Team, I’m facing an issue while fetching topic metadata from the Kafka cluster, resulting in the following exception. The metadata property is set to true in the ReadFromKafka KafkaIO transform: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

Re: ConsumerPollingTimeout

doc/current/apache_beam.io.kafka.html. > > On Fri, Sep 27, 2024 at 3:55 PM Utkarsh Parekh > wrote: > >> Hi Team, >> >> Is consumer polling timeout configurable for KafkaIO.read? >> >> Utkarsh >> >

ConsumerPollingTimeout

Hi Team, Is consumer polling timeout configurable for KafkaIO.read? Utkarsh

Re: Apache Bean on GCP / Forcing to use py 3.11

You have “mypackage” incorrectly built. Please check and confirm that. Utkarsh On Sun, Jun 16, 2024 at 12:48 PM Sofia’s World wrote: > Error is same...- see bottom - > i have tried to ssh in the container and the directory is setup as > expected.. so not quite sure where the issue is > i wi

Re: Offset access in Kafka messages in Python

HI Ondrej, You can retrieve metadata info using SDF (here is the video from Israel Herraiz) and the kafka-python PY package. Utkarsh On Tue, Mar 26, 2024 at 8:00 AM Ondřej Pánek wrote: > Hello team,

Re: Request to join slack channel

Hello, Can someone add me in slack channel as well? Thanks, Utkarsh On Wed, Feb 21, 2024 at 11:10 AM George Dekermenjian wrote: > Me too please - slack channel. > > > On Wed, Feb 21, 2024 at 19:43 Geddy Schellevis > wrote: > >> Hey Valentyn, >> >> can you add me to slack channel as well? >> T

Re: Beam 2.53.0 Release

Awesome!! On Fri, Jan 5, 2024 at 2:11 PM XQ Hu via user wrote: > Great! And thank you! > > On Fri, Jan 5, 2024 at 2:49 PM Jack McCluskey via user < > user@beam.apache.org> wrote: > >> We are happy to present the new 2.53.0 release of Beam. >> This release includes both improvements and new funct

Where to specify trust.jks

Hi, I'm testing a streaming app using kafka, Dafaflow, and Apache beam [Python]. "Error message from worker: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer org.apache.beam.sdk.util.UserCodeExcepti

Apache beam

Hi, I'm writing a simple streaming beam application. The application job is doing following tasks: 1. Reads data from GCS bucket (project 1) and loads into Kafka topic 2. Reads data from Kafka topic and loads into BigQUery (project 3) Composer running in Project 1 Data Flow running in project 2

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

And I also get this error occasionally when I execute a streaming pipeline with a new cluster instead of an existing cluster. https://issues.apache.org/jira/browse/BEAM-12032?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel On Tue, Feb 1, 2022 at 1:59 PM Utkarsh Parekh wrote

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

ould need to debug remotely the worker process. > > On 1 Feb 2022, at 19:18, Utkarsh Parekh > wrote: > > Sorry I sent the last message in a hurry. Here is the Beam java to kafka: > Is something missing here? > > > org.apache.beam > beam-sdks-java-io-kafka

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Sorry I sent the last message in a hurry. Here is the Beam java to kafka: Is something missing here? org.apache.beam beam-sdks-java-io-kafka 2.35.0 On Tue, Feb 1, 2022 at 9:01 AM Utkarsh Parekh wrote: > Here it is > > > org.apache.kafka > kafka-clie

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Here it is org.apache.kafka kafka-clients 2.8.0 On Tue, Feb 1, 2022 at 8:53 AM Alexey Romanenko wrote: > Hmm, this is strange. Which version of Kafka client do you use while > running it with Beam? > > On 1 Feb 2022, at 16:56, Utkarsh Parekh > wrote: > > Hi

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

est") .load() Utkarsh On Tue, Feb 1, 2022 at 6:20 AM Alexey Romanenko wrote: > Hi Utkarsh, > > Can it be related to this configuration problem? > > https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-troubleshooting-guide#no-records-received > > Did you check

Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hello, I'm doing POC with KafkaIO and spark runner on Azure Databricks. I'm trying to create a simple streaming app with Apache Beam, where it reads data from an Azure event hub and produces messages into another Azure event hub. I'm creating and running spark jobs on Azure Databricks. The probl