Apache Beam Write PubSub messages using Go

Hi Beam Community, I'm new to Apache Beam and trying to build a pipeline using Go. My requirement is to read a BigQuery table and publish the contents as a PubSub message. I couldn't find any information on the usage of pubsubio.Write function in the Go module. I want to know how to convert a PCo

Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi All, I'm new to using Apache Beam using Go. pubsubio.Write(scope, "project", "topic", ppMessages) When I try to publish a message in a topic I get an error message "Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1 I found a StackOverFlow post for the sa

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi All, Just sending a reminder in case anyone could help. I haven't received any response to my issue. Regards, Ashok On Fri, Feb 3, 2023 at 12:23 AM Ashok KS wrote: > Hi All, > > I'm new to using Apache Beam using Go. > > pubsubio.Write(scope, "project", &

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

Hi Shivam, Thanks a lot for your response. Yes it is a batch pipeline. My task is to read a big query table, process the data and publish the Rows as a PubSub message. Regards, Ashok On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal wrote: > Hey Ashok KS, > > Is this a batch pipeline? &

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

the code comments. Check the below issue for the > details: > https://github.com/apache/beam/issues/25326 > > On Mon, 6 Feb 2023 at 17:26, Ashok KS wrote: > >> Hi Shivam, >> >> Thanks a lot for your response. Yes it is a batch pipeline. My task is to >>

Re: Go + Apache Beam GCP Dataflow: Could not find the sink for pubsub, Check that the sink library specifies alwayslink = 1

ble, the input is finite and in result, > your pipeline is a batch pipeline. > I am not sure there is a straightforward way where you can convert this > pipeline into a streaming pipeline. > > > On Mon, 6 Feb 2023 at 17:32, Ashok KS wrote: > >> Hi Shivam, >> >>

Launch Dataflow Flex Templates from Go

Hello Beam Community, I have written a Dataflow pipeline using Python SDK and I would be creating a Flex template with it. My task is to launch this Flex Template from Cloud Functions which would be in Go. I found the package below but couldn't find any sample. https://cloud.google.com/dataflow/

Re: Launch Dataflow Flex Templates from Go

a flex template file and the docker image built, you >> can use the Dataflow API to run the template. >> >> https://cloud.google.com/dataflow/docs/reference/rest >> >> >> On Wed, 15 Feb 2023 at 04:49, Ashok KS wrote: >> >>> Hello Beam Community, >>&