[jira] [Created] (FLINK-16499) Flink shaded hadoop could not work when Yarn timeline service is enabled

2020-03-08 Thread Yang Wang (Jira)
Yang Wang created FLINK-16499: - Summary: Flink shaded hadoop could not work when Yarn timeline service is enabled Key: FLINK-16499 URL: https://issues.apache.org/jira/browse/FLINK-16499 Project: Flink

Re: [DISCUSS] Use SET statement to set table config in Flink SQL and implement a unified SQL call method

2020-03-08 Thread tison
Hi Weike, Thanks for kicking off this discussion! I cannot agree more on the proposal for a universal sql() method. It confuses & annoys our users a lot to distinguish sqlUpdate/sqlQuery and even insertInto and so on. IIRC there is an ongoing FLIP[1] dealing with the problem. You can checkout to

Flink Kafka consumer auto-commit timeout

2020-03-08 Thread Rong Rong
Hi All, I would like to bring back this discussion which I saw multiple times in previous ML threads [1], but there seem to have no solution if checkpointing is disabled. All of these ML reported exceptions have one common pattern: > *INFO* org.apache.kafka.clients.consumer.internals.AbstractCoo

[DISCUSS] Use SET statement to set table config in Flink SQL and implement a unified SQL call method

2020-03-08 Thread DONG, Weike
Hi dev, Recently we have tested the brand-new SQLClient and Flink SQL module, and we are amazed at this simple way of programming for streaming data analysis. However, as far as I know, the SET command is only available in the SQL Client, but not in SQL API. Although we understand that developers

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-08 Thread Becket Qin
Thanks for the reply, tison and Yang, Regarding the public interface, is "-R/--remote" option the only change? Will the users also need to provide a remote location to upload and store the jars, and a list of jars as dependencies to be uploaded? It would be important that the public interface sec

[jira] [Created] (FLINK-16498) make Postgres table work end-2-end in Flink SQL with PostgresJDBCCatalog

2020-03-08 Thread Bowen Li (Jira)
Bowen Li created FLINK-16498: Summary: make Postgres table work end-2-end in Flink SQL with PostgresJDBCCatalog Key: FLINK-16498 URL: https://issues.apache.org/jira/browse/FLINK-16498 Project: Flink

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-08 Thread Yang Wang
Hi Becket, Thanks for jumping out and sharing your concerns. I second tison's answer and just make some additions. > job submission interface This FLIP will introduce an interface for running user `main()` on cluster, named as “ProgramDeployer”. However, it is not a public interface. It will be

[jira] [Created] (FLINK-16497) Improve default flush strategy for JDBC sink to make it work out-of-box

2020-03-08 Thread Jark Wu (Jira)
Jark Wu created FLINK-16497: --- Summary: Improve default flush strategy for JDBC sink to make it work out-of-box Key: FLINK-16497 URL: https://issues.apache.org/jira/browse/FLINK-16497 Project: Flink

[jira] [Created] (FLINK-16496) Improve default flush strategy for HBase sink to make it work out-of-box

2020-03-08 Thread Jark Wu (Jira)
Jark Wu created FLINK-16496: --- Summary: Improve default flush strategy for HBase sink to make it work out-of-box Key: FLINK-16496 URL: https://issues.apache.org/jira/browse/FLINK-16496 Project: Flink

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-08 Thread tison
Hi Becket, Thanks for your attention on FLIP-85! I answered your question inline. 1. What exactly the job submission interface will look like after this FLIP? The FLIP template has a Public Interface section but was removed from this FLIP. As Yang mentioned in this thread above: >From user pers

[DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

2020-03-08 Thread Hequn Cheng
Hi everyone, FLIP-58 adds the support for Python UDFs, but user-defined metrics have not been supported yet. With metrics, users can report and monitor the UDF status to get a deeper understanding of the execution, so in this FLIP, we want to support metrics for Python UDFs. Previously, Jincheng

[jira] [Created] (FLINK-16495) Improve default flush strategy for Elasticsearch sink to make it work out-of-box

2020-03-08 Thread Jark Wu (Jira)
Jark Wu created FLINK-16495: --- Summary: Improve default flush strategy for Elasticsearch sink to make it work out-of-box Key: FLINK-16495 URL: https://issues.apache.org/jira/browse/FLINK-16495 Project: Flink

Re: [VOTE] [FLIP-85] Flink Application Mode

2020-03-08 Thread zoudan
This is a very useful featrue! +1 (non-binding) Best, Dan Zou

[jira] [Created] (FLINK-16494) Use enum type instead of string type for KubernetesConfigOptions.CONTAINER_IMAGE_PULL_POLICY

2020-03-08 Thread Canbin Zheng (Jira)
Canbin Zheng created FLINK-16494: Summary: Use enum type instead of string type for KubernetesConfigOptions.CONTAINER_IMAGE_PULL_POLICY Key: FLINK-16494 URL: https://issues.apache.org/jira/browse/FLINK-16494

Re: [PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

2020-03-08 Thread tison
Hi guys, So far, there is a PR[1] that implements the proposal in this thread. I look forward to your reviews or start a vote if required. Best, tison. [1] https://github.com/apache/flink/pull/10526 tison 于2020年3月6日周五 下午4:11写道: > Hi, > > Thanks for your rapid replies! > > Correctly when the

[jira] [Created] (FLINK-16493) Use enum type instead of string type for KubernetesConfigOptions.REST_SERVICE_EXPOSED_TYPE

2020-03-08 Thread Canbin Zheng (Jira)
Canbin Zheng created FLINK-16493: Summary: Use enum type instead of string type for KubernetesConfigOptions.REST_SERVICE_EXPOSED_TYPE Key: FLINK-16493 URL: https://issues.apache.org/jira/browse/FLINK-16493

[jira] [Created] (FLINK-16492) Flink SQL can not read decimal in hive parquet

2020-03-08 Thread xingoo (Jira)
xingoo created FLINK-16492: -- Summary: Flink SQL can not read decimal in hive parquet Key: FLINK-16492 URL: https://issues.apache.org/jira/browse/FLINK-16492 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-08 Thread Becket Qin
Hi Peter and Kostas, Thanks for creating this FLIP. Moving the JobGraph compilation to the cluster makes a lot of sense to me. FLIP-40 had the exactly same idea, but is currently dormant and can probably be superseded by this FLIP. After reading the FLIP, I still have a few questions. 1. What exa

[ANNOUNCE] Weekly Community Update 2020/10

2020-03-08 Thread Konstantin Knauf
Dear community, happy to share this week's community update featuring a set of exciting proposals from very different areas of Apache Flink, e.g. * Initial GPU Support * Flink Development Blog * "LIKE" in "CREATE TABLE" and a bit more. Enjoy. Flink Development == * [deployment] In

Re: [DISCUSS] FLIP-111: Docker image unification

2020-03-08 Thread Konstantin Knauf
Hi Andrey, thanks a lot for this proposal. The variety of Docker files in the project has been causing quite some confusion. For the entrypoint, have you considered to also allow setting configuration via environment variables as in "docker run -e FLINK_REST_BIN_PORT=8081 ..."? This is quite comm

[jira] [Created] (FLINK-16491) Add compression support for ParquetAvroWriters

2020-03-08 Thread Yao Zhang (Jira)
Yao Zhang created FLINK-16491: - Summary: Add compression support for ParquetAvroWriters Key: FLINK-16491 URL: https://issues.apache.org/jira/browse/FLINK-16491 Project: Flink Issue Type: Improvem

[jira] [Created] (FLINK-16490) Add a minimal statefun Python SDK

2020-03-08 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-16490: Summary: Add a minimal statefun Python SDK Key: FLINK-16490 URL: https://issues.apache.org/jira/browse/FLINK-16490 Project: Flink Issue Type: Task