The error says "Can't set both the topic and the subscription": PubSub
subscribers read from a subscription, and messages sent to a topic are sent
to all subscriptions bound to this topic. That's why, when reading from
PubSub in Beam you can specify either a topic (then a new subscription to
this topic will be created specifically for this pipeline) or a
subscription (if you already have a subscription you want to use for this
pipeline). What are you trying to accomplish by specifying both at the same
time?
On Mon, Nov 13, 2017, 12:08 AM vicky fazlurrahman <
vicky.fazlurrah...@gmail.com> wrote:
>
> Hi all,
>
> I am using beam sdk 2.1.0, trying to run streaming pipeline, using
> dataflow-runner that consume pub sub stream with the following config :
>
> p.apply(PubsubIO.readStrings()
> .fromTopic("projects/my-gcp-project/topics/publisher-test")
>
> .fromSubscription("projects/my-gcp-project/subscriptions/subscibe-test"))
>
> The build is failed because it unable to set the topic and subscription
>
> [DEBUG] Setting accessibility to true in order to invoke main().
>
> Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner
> fromOptions
> INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
> from the classpath: will stage 111 files. Enable logging at DEBUG level to
> see which files will be staged.
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Can't set both the topic and
> the subscription for a PubsubIO.Read transform
> at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:702)
> at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:536)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
> at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
> at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165)
> at com.example.drain.DrainPubsub.main(DrainPubsub.java:67)
> ... 6 more
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
>
> A bit confused, about the cause of the exception and what is the proper
> config for PubsubIO.
> Because seems, it can't get topic and subscription although given the
> correct format :
>
> projects/<project_id>/topics/<topic_name>
>
> projects/<project_id>/subscriptions/<subscription_name>
>
> Any help would be appreciated.
>
> --
> Vicky
>
