It depends on the input source: it will decide if your pipeline is a streaming or a batch pipeline.
Since you are querying over a BQ table, the input is finite and in result, your pipeline is a batch pipeline. I am not sure there is a straightforward way where you can convert this pipeline into a streaming pipeline. On Mon, 6 Feb 2023 at 17:32, Ashok KS <ashok7...@gmail.com> wrote: > Hi Shivam, > > Thanks for that. How can run the pipeline as a streaming pipeline? In > python I could just run the pipeline by passing —streaming=True in the > command line, but I couldn’t find anything similar in Go. > > Any pointers would be appreciated. > > Regards, > Ashok > > On Mon, 6 Feb 2023 at 10:59 pm, Shivam Singhal < > shivamsinghal5...@gmail.com> wrote: > >> The issue is not yet verified by the maintainers but I think the pubsubio >> connector's Write method doesn't work in Batch pipelines. >> >> But I am pretty sure that pubsubio Write doesn't work for Batch Pipelines >> because it's mentioned in the code comments. Check the below issue for the >> details: >> https://github.com/apache/beam/issues/25326 >> >> On Mon, 6 Feb 2023 at 17:26, Ashok KS <ashok7...@gmail.com> wrote: >> >>> Hi Shivam, >>> >>> Thanks a lot for your response. Yes it is a batch pipeline. My task is >>> to read a big query table, process the data and publish the Rows as a >>> PubSub message. >>> >>> Regards, >>> Ashok >>> >>> On Mon, 6 Feb 2023 at 10:52 pm, Shivam Singhal < >>> shivamsinghal5...@gmail.com> wrote: >>> >>>> Hey Ashok KS, >>>> >>>> Is this a batch pipeline? >>>> >>>> On Mon, 6 Feb 2023 at 09:27, Ashok KS <ashok7...@gmail.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Just sending a reminder in case anyone could help. I haven't received >>>>> any response to my issue. >>>>> >>>>> Regards, >>>>> Ashok >>>>> >>>>> On Fri, Feb 3, 2023 at 12:23 AM Ashok KS <ashok7...@gmail.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I'm new to using Apache Beam using Go. >>>>>> >>>>>> pubsubio.Write(scope, "project", "topic", ppMessages) >>>>>> When I try to publish a message in a topic I get an error message >>>>>> "Could not find the sink for pubsub, Check that the sink library >>>>>> specifies alwayslink = 1 >>>>>> >>>>>> I found a StackOverFlow post for the same issue but it doesn't solve >>>>>> the problem. >>>>>> >>>>>> Stackoverflow Link >>>>>> <https://stackoverflow.com/questions/69651665/go-apache-beam-gcp-dataflow-could-not-find-the-sink-for-pubsub-check-that-th> >>>>>> >>>>>> Can someone please help? >>>>>> >>>>>> Regards, >>>>>> Ashok >>>>>> >>>>>
