I already have a subscription for a pub sub topic. I just want to read from the created subscription. I See. My mistake. Its because i set both of fromTopic and fromSubscription at the same time.
Thank you very much Eugene. Regards, Vicky On Mon, Nov 13, 2017 at 3:58 PM, Eugene Kirpichov <[email protected]> wrote: > The error says "Can't set both the topic and the subscription": PubSub > subscribers read from a subscription, and messages sent to a topic are sent > to all subscriptions bound to this topic. That's why, when reading from > PubSub in Beam you can specify either a topic (then a new subscription to > this topic will be created specifically for this pipeline) or a > subscription (if you already have a subscription you want to use for this > pipeline). What are you trying to accomplish by specifying both at the same > time? > > On Mon, Nov 13, 2017, 12:08 AM vicky fazlurrahman < > [email protected]> wrote: > >> >> Hi all, >> >> I am using beam sdk 2.1.0, trying to run streaming pipeline, using >> dataflow-runner that consume pub sub stream with the following config : >> >> p.apply(PubsubIO.readStrings() >> .fromTopic("projects/my-gcp-project/topics/publisher-test") >> .fromSubscription("projects/my-gcp-project/subscriptions/ >> subscibe-test")) >> >> The build is failed because it unable to set the topic and subscription >> >> [DEBUG] Setting accessibility to true in order to invoke main(). >> >> Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner >> fromOptions >> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files >> from the classpath: will stage 111 files. Enable logging at DEBUG level to >> see which files will be staged. >> [WARNING] >> java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke( >> NativeMethodAccessorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at org.codehaus.mojo.exec.ExecJavaMojo$1.run( >> ExecJavaMojo.java:293) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.IllegalStateException: Can't set both the topic and >> the subscription for a PubsubIO.Read transform >> at org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand( >> PubsubIO.java:702) >> at org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand( >> PubsubIO.java:536) >> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514) >> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454) >> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) >> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165) >> at com.example.drain.DrainPubsub.main(DrainPubsub.java:67) >> ... 6 more >> [INFO] ------------------------------------------------------------ >> ------------ >> [INFO] BUILD FAILURE >> [INFO] ------------------------------------------------------------ >> ------------ >> >> A bit confused, about the cause of the exception and what is the proper >> config for PubsubIO. >> Because seems, it can't get topic and subscription although given the >> correct format : >> >> projects/<project_id>/topics/<topic_name> >> >> projects/<project_id>/subscriptions/<subscription_name> >> >> Any help would be appreciated. >> >> -- >> Vicky >> > -- Best Regards, *Vicky Fazlurrahman* *GO-JEK INDONESIA* E-mail [email protected] Website www.go-jek.com Phone +62 85729516006 <+62+85729516006>
