[
https://issues.apache.org/jira/browse/BEAM-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133365#comment-17133365
]
Brian Hulette edited comment on BEAM-9208 at 6/11/20, 4:09 PM:
---------------------------------------------------------------
The syntax proposed here would be difficult to implement since it will require
changing the DDL parser. We could also just put this metadata in TBLPROPERTIES
(this would also be more compatible with the proposal in
https://s.apache.org/beam-schema-io which doesn't discuss column-level
metadata):
{code}
CREATE TABLE people (
my_timestamp TIMESTAMP,
my_id VARCHAR,
name VARCHAR,
age INTEGER
)
TYPE 'pubsub'
LOCATION 'projects/my-project/topics/my-topic'
TBLPROPERTIES {
"columnMappings": {
"my_timestamp": "pubsub:event_timestamp"
"my_id": "pubsub:attributes[id_name]"
}
}
{code}
was (Author: bhulette):
The syntax proposed here would be difficult to implement since it will require
changing the DDL parser. We could also just put this metadata in TBLPROPERTIES
(this would also be more compatible with the proposal in
https://s.apache.org/beam-schema-io):
{code}
CREATE TABLE people (
my_timestamp TIMESTAMP,
my_id VARCHAR,
name VARCHAR,
age INTEGER
)
TYPE 'pubsub'
LOCATION 'projects/my-project/topics/my-topic'
TBLPROPERTIES {
"columnMappings": {
"my_timestamp": "pubsub:event_timestamp"
"my_id": "pubsub:attributes[id_name]"
}
}
{code}
> Add support for mapping columns to pubsub message attributes in flat schemas
> DDL
> --------------------------------------------------------------------------------
>
> Key: BEAM-9208
> URL: https://issues.apache.org/jira/browse/BEAM-9208
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Brian Hulette
> Priority: P2
>
> Context:
> https://lists.apache.org/thread.html/bf4c37f21bda194d7f8c40f6e7b9a776262415755cc1658412af3c76%40%3Cdev.beam.apache.org%3E
> The syntax should look something like this (proposed by [~alexvanboxel]):
> {code}
> CREATE TABLE people (
> my_timestamp TIMESTAMP *OPTION(ref="pubsub:event_timestamp)*,
> my_id VARCHAR *OPTION(ref="pubsub:attributes['id_name']")*,
> name VARCHAR,
> age INTEGER
> )
> TYPE 'pubsub'
> LOCATION 'projects/my-project/topics/my-topic'
> {code}
> This jira pertains specifically to the my_id field in this example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
