[jira] [Created] (FLINK-14162) Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies

Zhu Zhu created FLINK-14162: --- Summary: Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies Key: FLINK-14162 URL: https://issues.apache.org/jira/browse/FLINK-14162 P

[jira] [Created] (FLINK-14163) A Task should be deployed only after all its partitions have completed the registration

Zhu Zhu created FLINK-14163: --- Summary: A Task should be deployed only after all its partitions have completed the registration Key: FLINK-14163 URL: https://issues.apache.org/jira/browse/FLINK-14163 Project

Re: Re: [DISCUSS] Support JSON functions in Flink SQL

Hi Jack, Thank you very much for your reply, google doc I have updated, and some of your questions I replied. In addition, I want to apply for Flip permissions for this purpose. Best, Forward Jark Wu 于2019年9月20日周五 下午9:53写道： > Hi Forward, > > Sorry for the late reply. I have went through the de

Confluence permission for FLIP creation

Hi devs, I'd like to create a page about the Support SQL 2016-2017 JSON functions in Flink SQL FLIP. Could you grant me Confluence permission for FLIP creation? My Confluence

Re: non-deserializable root cause in DeclineCheckpoint

Hi, Jeffrey~ I think two fixes you mentioned may not work in your case. This problem https://issues.apache.org/jira/browse/FLINK-14076 is caused by TM and JM jar package environment inconsistent or jar loaded behavior inconsistent in nature. M

Re: Best coding practises guide while programming using flink apis

Hi, Deepak~ I appreciate your idea and cc to dev mail too. Best, Terry Wang > 在 2019年9月22日，下午2:12，Deepak Sharma 写道： > > Hi All > I guess we need to put some examples in the documentation around best coding > practises , concurrency , non blocking IO and design patterns while writing > Apa

Re: [DISCUSS] FLIP-48: Pluggable Intermediate Result Storage

## About the improvements you mentioned in (1) - I am not sure that this helps to improve performance by avoiding to break the pipeline. Attaching an additional sink, would in virtually any case add even more overhead than the pipeline breaking. What is your reasoning why it would be fas

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

My assumption is as Sijie's, that once the connector is either part of Flink, or part of the streamnative repo. No double maintenance. I feel this discussion is very much caught in problems that are all solvable if we want to solve them. Maybe we can think what our goal for users and the communiti

[DISCUSS] Add Bucket File System Connector

Hi,everyone:       In the current flink system, use flink sql to read data and then write it to File System with kind of formats is not supported, the current File System Connector is only experimental [1], so I have developed a new File System Connector.        Thanks to the suggestion of Kurt

use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in Flink Sql client

Hi ,   Thanks again for responding on my earlier queries..     I was again going through the Flink SQL client code and came across the default custom command-line , few days back i came to know that Flink sql client is not supported in a full fledged cluster with different resource managers like

[ANNOUNCE] Weekly Community Update 2019/38

Dear community, happy to share this week's community update with a FLIP for the Pulsar Connector contribution, three FLIPs for the SQL Ecosystem (plugin system, computed columns, extended support for views), and a bit more. Enjoy! Flink Development == * [connectors] After some discu

Re: non-deserializable root cause in DeclineCheckpoint

Thanks for suggestion, Terry. I've investigated a bit further. DeclineCheckpoint specifically checks for the possibility of an exception that the JM won't be able to deserialize (i.e. anything other than a Checkpoint exception). It just doesn't check for the possibility of a CheckpointException th

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Stephan, I have no doubt about the value of adding Pulsar connector to Flink repo. My concern is about how exactly we are going to do it. As mentioned before, I believe that we can handle connectors more > pragmatically and less strict than the core of Flink, if it helps unlocking > users. I

Re: non-deserializable root cause in DeclineCheckpoint

Hi, Jeffrey~ Thanks for your detailed explanation and I understood why job failed with flink 1.9. But the two fixes you mentioned may still not work well. As KafkaException can be serialized in TM for there is necessary jar in its classpath but not in JM, so maybe it’s impossible to check the

Re: use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in Flink Sql client

Hi Dipanjan: I just looked through the Flink SQL client code and got the same conclusion as you. Look forward to receiving other comments. Best, Terry Wang > 在 2019年9月22日，下午11:53，Dipanjan Mazumder 写道： > > Hi , > Thanks again for responding on my earlier queries.. > I was again going t

Re: [SURVEY] How many people are using customized RestartStrategy(s)

Steven, Thanks for the information. If we can determine this a common issue, we can solve it in Flink core. To get to that state, I have two questions which need your help: 1. Why is gauge not good for alerting? The metric "fullRestart" is a Gauge. Does the metric reporter you use report Counter a

Re: How to prevent from launching 2 jobs at the same time

The situation is as Dian said. Flink identifies jobs by job id instead of job name. However, I think it is still a valid question if it is an alternative Flink identifies jobs by job name and leaves the work to distinguish jobs by name to users. The advantages in this way includes a readable displ

[jira] [Created] (FLINK-14164) Add a metric to show failover count regarding fine grained recovery

Zhu Zhu created FLINK-14164: --- Summary: Add a metric to show failover count regarding fine grained recovery Key: FLINK-14164 URL: https://issues.apache.org/jira/browse/FLINK-14164 Project: Flink Is

Re: non-deserializable root cause in DeclineCheckpoint

Hi Terry, KafkaException comes in through the job's dependencies (it's defined in the kafka-clients jar packed up in the fat job jar) and is on either the TM nor JM default classpath. The job running in the TM includes the job dependencies and so can throw a KafkaException but the JM can't deseria

Re: [DISCUSS] Add Bucket File System Connector

Hi jun: Sorry for the late reply, I share my thoughts on StreamingFileSink in FLIP-63 [1] and I don't recommend using StreamingFileSink to support partitioning in Table. 1.The bucket concept and SQL's bucket concept are in serious conflict.[2] 2.In table, we need support single-partition writin

Re: non-deserializable root cause in DeclineCheckpoint

Hi Jeffrey, You are right and I understood what you have said after I just studied the class org.apache.flink.util.SerializedThrowable. I prefer the fixes no.2 you mentioned: CheckpointException should always capture its wrapped exception as a SerializedThrowable Looking forward to se

Per Key Grained Watermark Support

Hi all, Currently Watermark can only be supported on task’s level(or partition level), which means that the data belonging to the faster key has to share the same watermark with the data belonging to the slower key in the same key group of a KeyedStream. This will lead to two problems: 1. L

Re: use of org.apache.flink.yarn.cli.FlinkYarnSessionCli in Flink Sql client

Hi Dipanjan, I think you are right that it's already been supported to submit a job to cluster via SQL client. This is supported in [1]. Besides, I think that it is not configured in the YAML. It's specified in the CLI options when you start up the SQL client CLI. [1] https://issues.apache.org

Re: Per Key Grained Watermark Support

Hi Jiayi We have face the same challenge as we deal with IoT unit and they do not necessarily share the same timestamp. Watermark or. Key would be perfect match here. We tried to workaround with handle late events as special case with sideoutputs but isn’t the perfect solution. My conclusion i

[DISCUSS] Releasing Flink 1.9.1

Hi everyone, It has already been a month since we released Flink 1.9.0. We already have many important bug fixes from which our users can benefit in the release-1.9 branch (83 resolved issues). Therefore, I propose to create the next bug fix release for Flink 1.9. Most notable fixes are: - [FLIN

Re: [DISCUSS] Releasing Flink 1.9.1

FLINK-13708 is also very critical IMO. This would cause invalid flink job (doubled output) https://issues.apache.org/jira/browse/FLINK-13708 Jark Wu 于2019年9月23日周一 下午2:03写道： > Hi everyone, > > It has already been a month since we released Flink 1.9.0. > We already have many important bug fixes f

Re: [DISCUSS] Releasing Flink 1.9.1

I hope https://issues.apache.org/jira/browse/FLINK-12501 will also be part of 1.9.1 .. regards. On Mon, Sep 23, 2019 at 11:39 AM Jeff Zhang wrote: > FLINK-13708 is also very critical IMO. This would cause invalid flink job > (doubled output) > > https://issues.apache.org/jira/browse/FLINK-13708

Re: non-deserializable root cause in DeclineCheckpoint

Draft PR here: https://github.com/apache/flink/pull/9742 There might be some failing tests (still waiting on Travis), but I think the diff is small enough for you to evaluate the approach for acceptability. On Sun, Sep 22, 2019 at 9:10 PM Terry Wang wrote: > Hi Jeffrey, > > You are right and I u