Influxdb reporter not honouring the metrics scope

2020-01-19 Thread Gaurav Singhania
Hi, We are using influxdb reporter for flink 1.9 to capture our metrics. We want to override the scope of task metrics, however even after providing the config in yaml file the metrics continues to have the tags we don't want. The metric scope we want to change is : *metrics.scope.task *with a def

Re: flink1.9.1 RetractStream insert to mysql problem

2020-01-19 Thread Jingsong Li
Hi Polarisary, Look at the semantics your SQL wants to express: Top N, More practically is: Top 1. - Top N produce stream with primary keys contains row number, but your sql didn't select row number, so there is not primary key. - UpsertStreamTableSink requires primary key, So there is an exceptio

flink1.9.1 RetractStream insert to mysql problem

2020-01-19 Thread Polarisary
Hi, I use flink 1.9.1, sql as follows, INSERT INTO a SELECT c1, c2, c3, c4 FROM ( SELECT *,ROW_NUMBER() OVER (PARTITION BY c1, c2, c3 ORDER BY c4 DESC) AS rownum" + FROM t)

Re: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

2020-01-19 Thread Yun Tang
Hi Lake A more suitable place for this mail should be in user-mail list. There are three reasons why this could happen: 1. This file is orphan file e.g. file uploaded during one checkpoint but task manager exited unexpectedly leave that checkpoint not completed. 2. This file should be rem

[ANNOUNCE] Weekly Community Update 2020/03

2020-01-19 Thread Konstantin Knauf
Dear community, happy to share this week's weekly community digest with a release candidate for Flink 1.10, a Pulsar Catalog for Flink, a 50% discount code for Flink Forward SF and bit more. Flink Development == * [releases] The first (preview)* release candidate for Flink 1.10* has

Re: PubSub source throwing grpc errors

2020-01-19 Thread Itamar Syn-Hershko
Definitely (though will be hijacking my own thread). The root issue is that it seems impossible to load custom Jackson Modules, specifically in our case JodaModule and KotlinModule . Since Jack

Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

2020-01-19 Thread LakeShen
Hi community, now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir : 1. chk -xx dir 2. shared dir 3. taskowned dir I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest checkpo

[DISCUSS] Active Kubernetes integration phase2

2020-01-19 Thread Yang Wang
Hi everyone, Currently Flink supports the resource management system YARN and Mesos. However, they were not designed for fast moving cloud native architectures, and they could not support mixed workloads (e.g. batch, streaming, deep learning, web services, etc.) relatively well. At the same time

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

2020-01-19 Thread kant kodali
I tried the following. bsTableEnv.sqlUpdate("CREATE VIEW my_view AS SELECT * FROM sample1 FULL OUTER JOIN sample2 on sample1.f0=sample2.f0"); Table result = bsTableEnv.sqlQuery("select * from my_view"); It looks like https://cwiki.apache.org/confluence/display/FLINK/FLIP-71+-+E2E+View+support+i