Hello Everyone, *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. I remember what it was like first learning Beam on my own and I am passionate about everyone's learning experience. Below are definitions/references and a suggested learning path to understand this email.*
*Short Version (assumes Beam knowledge)*: I submitted the last of planned pull requests [1] that allows Pub/Sub [2] Schema Transform [3] support. What remains is increasing integration test coverage [4]. *Long Version (for those first learning Beam)*: I'm going to explain this without using Beam specific language. Suppose we need to publish data to a Pub/Sub topic. Instead of writing code to do this, what if we could send some backend service a single configuration file that specifies what the Pub/Sub topic is and the format of our data such as JSON or Avro? Let the service do everything else and determine how to get our data into Pub/Sub. *Definitions/References*: 1. https://github.com/apache/beam/pull/24443 2. https://cloud.google.com/pubsub/docs/overview 3. *Schema Transform* - Enables Schema Aware transforms when reading from and writing to sources and sinks using Beam IOs defined declaratively using a single configuration input 4. https://github.com/apache/beam/issues/24462 5. *Schema Aware* - Schema Awareness refers to transforms that know how to process pipeline elements with inherent knowledge of their properties and types. This collection of properties and types is called a Schema. A Beam Row contains properties and a data structure described by a Schema. Think of the Row as the data element described by its Schema. *Suggested Learning Path To Understand This Email*: 1. https://cloud.google.com/pubsub/docs/publisher 2. https://beam.apache.org/documentation/programming-guide/#overview 3. https://beam.apache.org/documentation/programming-guide/#pcollection-characteristics 4. https://beam.apache.org/documentation/programming-guide/#transforms (Up to 4.1) 5. https://beam.apache.org/documentation/programming-guide/#schemas Best, Damon
