Good morning,
I'm very new to Beam, and pretty new to Python so please first accept my
apologies for any obvious misconceptions/mistakes in the following.
I am currently trying to develop a sample pipeline in Python to pull messages
from Pub/Sub and then write them to either files in cloud storage or to
BigQuery. The ultimate goal will be to utilise the pipeline for real time
streaming of event data to BigQuery (with various transformations) but also to
store the raw messages long term in files in cloud storage.
At the moment, I'm simply trying to parse the message to get the PubSub
messageId and publishTime in order to be able to write them into the output.
The json of my PubSub message looks like this:-
[
{
"ackId":
"BCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIICBQFfH1xU1t1Xl8aB1ENGXJ8Zyc_XxcIB0BTeFVaEQx6bVxXOFcMEHF8YXZpWhUIA0FTfXeq5cveluzJNksxIbvE8KxfeqqmgfhiZho9XxJLLD5-PT5FQV5AEkw2C0RJUytDCypYEU4",
"message": {
"attributes": {
"source": "python"
},
"data": "eyJyb3dudW1iZXIiOiAyfQ==",
"messageId": "619310330691403",
"publishTime": "2019-07-12T08:27:58.522Z"
}
}
]
According to the
documentation<https://beam.apache.org/releases/pydoc/2.13.0/apache_beam.io.gcp.pubsub.html>
the PubSub message payload returns the data and attributes properties; is
there simply no way of retrieving the messageId and publishTime, or are these
exposed somewhere else? If not, will the inclusion of these be in the roadmap,
and are they available if using Java (I have zero Java experience hence why
reaching for Python first).
Kind regards,
Matthew
Disclaimer
The information contained in this communication from the sender is
confidential. It is intended solely for use by the recipient and others
authorized to receive it. If you are not the recipient, you are hereby notified
that any disclosure, copying, distribution or taking action in relation of the
contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been
automatically archived by Mimecast Ltd, an innovator in Software as a Service
(SaaS) for business. Providing a safer and more useful place for your human
generated data. Specializing in; Security, archiving and compliance. To find
out more visit the Mimecast website.
