Airflow + beam + job: read from gs - write to pubsub

[Hartweger Günter]

Hi everyone,


I do have a question, is it possible to run beam jobs scheduled by Airflow that 
reads from
Txt file from google storage and write to pubsub? If yes need some running 
example.

Here is what I did:
(p
            | "Read from gs" >> beam.io.ReadFromText('gs:// .... /test.txt')
            | 'write to pubsub' >> 
beam.io.WriteToPubSub("projects/<projectname> /topics/<topicname> ")
            )

Can run the job from dataflow with custom templet.

But if I run with Composer get this error: Workflow failed. Causes: Expected 
custom source to have non-zero number of splits.


Can anyone help?

Thanks,
GH