Hi all,
I am using beam sdk 2.1.0, trying to run streaming pipeline, using
dataflow-runner that consume pub sub stream with the following config :
p.apply(PubsubIO.readStrings()
.fromTopic("projects/my-gcp-project/topics/publisher-test")
.fromSubscription("projects/my-gcp-project/subscriptions/subscibe-test"))
The build is failed because it unable to set the topic and subscription
[DEBUG] Setting accessibility to true in order to invoke main().
Nov 13, 2017 2:23:03 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files
from the classpath: will stage 111 files. Enable logging at DEBUG level to
see which files will be staged.
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Can't set both the topic and
the subscription for a PubsubIO.Read transform
at
org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:702)
at
org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:536)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:165)
at com.example.drain.DrainPubsub.main(DrainPubsub.java:67)
... 6 more
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
A bit confused, about the cause of the exception and what is the proper
config for PubsubIO.
Because seems, it can't get topic and subscription although given the
correct format :
projects/<project_id>/topics/<topic_name>
projects/<project_id>/subscriptions/<subscription_name>
Any help would be appreciated.
--
Vicky
