[
https://issues.apache.org/jira/browse/FLINK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Kwon updated FLINK-19517:
-------------------------------
Description:
Currently, table creation from SQL client such as below works well
{code:sql}
CREATE TABLE kafkaTable (
user_id BIGINT,
item_id BIGINT,
category_id BIGINT,
behavior STRING,
ts TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'testGroup',
'format' = 'avro',
'scan.startup.mode' = 'earliest-offset'
)
{code}
Although I would wish for the table creation to support Confluent Kafka
configuration as well. For example something like
{code:sql}
CREATE TABLE kafkaTable (
user_id BIGINT,
item_id BIGINT,
category_id BIGINT,
behavior STRING,
ts TIMESTAMP(3)
) WITH (
'connector' = 'confluent-kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'testGroup',
'schema-registry' = 'http://schema-registry.com',
'scan.startup.mode' = 'earliest-offset'
)
{code}
If this is enabled, it will be much more convenient to test queries on-the-fly
that business analysts want to test against with 'Confluent Kafka'
Additionally, it will be better if we can
- specify 'parallelism' within WITH clause to support parallel partition
processing
- specify custom properties within WITH clause specified in
[https://docs.confluent.io/5.4.2/installation/configuration/consumer-configs.html]
- have remote access to SQL client in cluster from local environment
was:
Currently, table creation from SQL client such as below works well
{code:sql}
CREATE TABLE kafkaTable (
user_id BIGINT,
item_id BIGINT,
category_id BIGINT,
behavior STRING,
ts TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'testGroup',
'format' = 'avro',
'scan.startup.mode' = 'earliest-offset'
)
{code}
Although I would wish for the table creation to support Confluent Kafka
configuration as well. For example something like
{code:sql}
CREATE TABLE kafkaTable (
user_id BIGINT,
item_id BIGINT,
category_id BIGINT,
behavior STRING,
ts TIMESTAMP(3)
) WITH (
'connector' = 'confluent-kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'testGroup',
'schema-registry' = 'http://schema-registry.com',
'scan.startup.mode' = 'earliest-offset'
)
{code}
If this is enabled, it will be much more convenient to test queries on-the-fly
that business analysts want to test against with Confluent Kafka (which is
widely used). For example, Business analysts can give some standard DDL to data
engineers and data engineers can fill in the WITH clause and immediately start
executing queries against these tables
Additionally, it will be better if we can
- specify 'parallelism' within WITH clause to support parallel partition
processing
- specify custom properties within WITH clause specified in
[https://docs.confluent.io/5.4.2/installation/configuration/consumer-configs.html]
- have remote access to SQL client in cluster from local environment
> Support for Confluent Kafka of Table Creation in Flink SQL Client
> -----------------------------------------------------------------
>
> Key: FLINK-19517
> URL: https://issues.apache.org/jira/browse/FLINK-19517
> Project: Flink
> Issue Type: Wish
> Affects Versions: 1.12.0
> Reporter: Kevin Kwon
> Priority: Critical
>
> Currently, table creation from SQL client such as below works well
> {code:sql}
> CREATE TABLE kafkaTable (
> user_id BIGINT,
> item_id BIGINT,
> category_id BIGINT,
> behavior STRING,
> ts TIMESTAMP(3)
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behavior',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'testGroup',
> 'format' = 'avro',
> 'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> Although I would wish for the table creation to support Confluent Kafka
> configuration as well. For example something like
> {code:sql}
> CREATE TABLE kafkaTable (
> user_id BIGINT,
> item_id BIGINT,
> category_id BIGINT,
> behavior STRING,
> ts TIMESTAMP(3)
> ) WITH (
> 'connector' = 'confluent-kafka',
> 'topic' = 'user_behavior',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'testGroup',
> 'schema-registry' = 'http://schema-registry.com',
> 'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> If this is enabled, it will be much more convenient to test queries
> on-the-fly that business analysts want to test against with 'Confluent Kafka'
> Additionally, it will be better if we can
> - specify 'parallelism' within WITH clause to support parallel partition
> processing
> - specify custom properties within WITH clause specified in
> [https://docs.confluent.io/5.4.2/installation/configuration/consumer-configs.html]
> - have remote access to SQL client in cluster from local environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
