Re: JobManager scale limitation - Slow S3 checkpoint deletes

Sharing the communication pressure of a single node to multi task managers would be a good idea. From my point of view, let task managers to know the information that some specific checkpoint had already been aborted could benefit a lot of things: * Let task manager to clean up the files, w

[jira] [Created] (FLINK-11848) Delete outdated kafka topics caused UNKNOWN_TOPIC_EXCEPTIION

Shengnan YU created FLINK-11848: --- Summary: Delete outdated kafka topics caused UNKNOWN_TOPIC_EXCEPTIION Key: FLINK-11848 URL: https://issues.apache.org/jira/browse/FLINK-11848 Project: Flink I

Re: JobManager scale limitation - Slow S3 checkpoint deletes

Nice! Perhaps for file systems without TTL/expiration support (AFAIK includes HDFS), cleanup could be performed in the task managers? On Wed, Mar 6, 2019 at 6:01 PM Jamie Grier wrote: > Yup, it looks like the actor threads are spending all of their time > communicating with S3. I've attached

Re: JobManager scale limitation - Slow S3 checkpoint deletes

Yup, it looks like the actor threads are spending all of their time communicating with S3. I've attached a picture of a typical stack trace for one of the actor threads [1]. At the end of that call stack what you'll see is the thread blocking on synchronous communication with the S3 service. Thi

Fwd: DataStream EventTime last data cannot be output？

> 下面是被转发的邮件： > > 发件人: 刘 文 > 主题: DataStream EventTime last data cannot be output？ > 日期: 2019年3月6日 GMT+8 下午10:51:14 > 收件人: u...@flink.apache.org > > DataStream EventTime last data cannot be output ？ > > > In the verification of EventTime plus watermark processing, I found that the > data sent

Re: [ANNOUNCEMENT] March 2019 Bay Area Apache Flink Meetup

Hi all, This is a kind reminder that our next Flink meetup will be a couple of weeks away. This is the opportunity to share experience or gain insights, or just get socialized in the community. RSVP is required, which can be done at the meetup webpage

Re: [DISCUSS] Create a Flink ecosystem website

I like Shaoxuan's idea to keep this a static site first. We could then iterate on this and make it a dynamic thing. Of course, if we have the resources in the community to quickly start with a dynamic site, I'm not apposed. – Ufuk On Wed, Mar 6, 2019 at 2:31 PM Robert Metzger wrote: > > Awesome!

Re: [DISCUSS] FLIP-33: Terminate/Suspend Job with Savepoint

I really like this effort. I think the original plan for "cancel-with-savepoint" was always to just be a workaround until we arrived at a better solution as proposed here. Regarding the FLIP, I agree with Elias comments. I think the number of termination modes the FLIP introduces can be overwhelmi

[jira] [Created] (FLINK-11847) Docker job-specific image creation instruction make reference to --job-jar parameter that does not actually exist in the script

Frank Wilson created FLINK-11847: Summary: Docker job-specific image creation instruction make reference to --job-jar parameter that does not actually exist in the script Key: FLINK-11847 URL: https://issues.apach

[jira] [Created] (FLINK-11846) Duplicate job submission delete HA files

Till Rohrmann created FLINK-11846: - Summary: Duplicate job submission delete HA files Key: FLINK-11846 URL: https://issues.apache.org/jira/browse/FLINK-11846 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-11845) Drop unused InfoMessage

TisonKun created FLINK-11845: Summary: Drop unused InfoMessage Key: FLINK-11845 URL: https://issues.apache.org/jira/browse/FLINK-11845 Project: Flink Issue Type: Improvement Components:

[jira] [Created] (FLINK-11844) Simplify OVER window Table API classes

Timo Walther created FLINK-11844: Summary: Simplify OVER window Table API classes Key: FLINK-11844 URL: https://issues.apache.org/jira/browse/FLINK-11844 Project: Flink Issue Type: Improvemen

Re: JobManager scale limitation - Slow S3 checkpoint deletes

I think having an option to not actively delete checkpoints (but rather have the TTL feature of the file system take care of it) sounds like a good idea. I am curious why you get heartbeat misses and akka timeouts during deletes. Are some parts of the deletes happening sychronously in the actor th

[jira] [Created] (FLINK-11843) Dispatcher fails to recover jobs if leader change happens during JobManagerRunner termination

Till Rohrmann created FLINK-11843: - Summary: Dispatcher fails to recover jobs if leader change happens during JobManagerRunner termination Key: FLINK-11843 URL: https://issues.apache.org/jira/browse/FLINK-11843

Re: [DISCUSS] Using Guava in Flink "core" packages

How I managed to do that.. Here is the discussion about the shared package: https://lists.apache.org/thread.html/3de9d2353cf22aea0448fb744314103b5f88195216acc3bff449354a@%3Cdev.flink.apache.org%3E On Wed, Mar 6, 2019 at 7:10 AM Aljoscha Krettek wrote: > I think the two links are identical. >

Re: [DISCUSS] Using Guava in Flink "core" packages

I think the two links are identical. > On 6. Mar 2019, at 16:05, Thomas Weise wrote: > > For more context, see [1] [2] > > The GuavaFlinkConnectorRateLimiter is an implementation of the rate limiter > interface that uses Guava. It is not a test class, it is intended to be > used in applications

Re: [DISCUSS] Using Guava in Flink "core" packages

For more context, see [1] [2] The GuavaFlinkConnectorRateLimiter is an implementation of the rate limiter interface that uses Guava. It is not a test class, it is intended to be used in applications and the (shaded) Guava isn't user facing. [1] https://lists.apache.org/thread.html/3599d95020604e2

Re: [DISCUSS] Using Guava in Flink "core" packages

Hi, yes I also fully agree that it is time to write down all these implicit convensions that we've learned throught the last years. The Flink community is growing quite rapidly right now and we must ensure that the same mistakes do not repeat again. Keeping the number of dependencies low is

Re: [DISCUSS] Using Guava in Flink "core" packages

I fully agree that we should write down this kind of conventions that committers have setup implicitly. I agree that we should keep flink-core as lean as possible. On guava usages in general my current stance is that we should only use guava if necessary. Spreading it (or other dependencies fo

JobManager scale limitation - Slow S3 checkpoint deletes

We've run into an issue that limits the max parallelism of jobs we can run and what it seems to boil down to is that the JobManager becomes unresponsive while essentially spending all of it's time discarding checkpoints from S3. This results in sluggish UI, sporadic AkkaAskTimeouts, heartbeat miss

[DISCUSS] Using Guava in Flink "core" packages

Hi, I recently saw that we added a dependency on our shaded-guava to flink-core [1]. Just for the record, I don’t want do diminish the contributions of anyone involved in the PR in any way. It just made me realise that we have some implicit agreements or assumptions about adding certain things

4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las Vegas, and Berlin!

Dear Apache Enthusiast, (You’re receiving this because you are subscribed to one or more user mailing lists for an Apache Software Foundation project.) TL;DR: * Apache Roadshow DC is in 3 weeks. Register now at https://apachecon.com/usroadshowdc19/ * Registration for Apache Roadshow Chicago is

[jira] [Created] (FLINK-11842) Introduce FlinkLogicalMatch

godfrey he created FLINK-11842: -- Summary: Introduce FlinkLogicalMatch Key: FLINK-11842 URL: https://issues.apache.org/jira/browse/FLINK-11842 Project: Flink Issue Type: New Feature Com

[jira] [Created] (FLINK-11841) Introduce FlinkLogicalSemiJoin

godfrey he created FLINK-11841: -- Summary: Introduce FlinkLogicalSemiJoin Key: FLINK-11841 URL: https://issues.apache.org/jira/browse/FLINK-11841 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Create a Flink ecosystem website

Awesome! Thanks a lot for looking into this Becket! The VMs hosted by Infra look suitable. @Shaoxuan: There is actually already a static page. It used to be linked, but has been removed from the navigation bar for some reason. This is the page: https://flink.apache.org/ecosystem.html We could upda

[jira] [Created] (FLINK-11840) Introduce FlinkLogicalSnapshot

godfrey he created FLINK-11840: -- Summary: Introduce FlinkLogicalSnapshot Key: FLINK-11840 URL: https://issues.apache.org/jira/browse/FLINK-11840 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-11839) Introduce the rest flink logical relational nodes

godfrey he created FLINK-11839: -- Summary: Introduce the rest flink logical relational nodes Key: FLINK-11839 URL: https://issues.apache.org/jira/browse/FLINK-11839 Project: Flink Issue Type: New

Re: [DISCUSS] Improve the flinkbot

This is the picture: https://user-images.githubusercontent.com/89049/53882383-7fda9380-4016-11e9-877d-10cdc00bdfbd.png Speaking about feature requests, priorities and time-spend: My plan was to now work on introducing a new label category for the components. This should get us a lot better overvie

[jira] [Created] (FLINK-11838) Create RecoverableWriter for GCS

Fokko Driesprong created FLINK-11838: Summary: Create RecoverableWriter for GCS Key: FLINK-11838 URL: https://issues.apache.org/jira/browse/FLINK-11838 Project: Flink Issue Type: Improvem

Re: Error on importing flink source code into Intellij

And sometimes just reimport maven will work. Right click pom.xml located in Flink's root dir -> Maven -> Reimport Best, Kurt On Wed, Mar 6, 2019 at 8:02 PM Chesnay Schepler wrote: > Usually when I run into this i use "File -> Invalidate Caches / > Restart... -> Invalidate and restart" > > On

Re: Contributor permission application

You now have contributor permissions. On 05.03.2019 04:57, zenppl liu wrote: Hi, Sorry for the wrong ID. My username is zenppl. Thank you, Zenppl 在 2019年3月4日，下午6:34，Fabian Hueske 写道： Hi, Welcome to the Flink community. Jira does not use the email address as user name. Hence I couldn't add y

Re: WELCOME to dev@flink.apache.org

I've given you contributor permissions. On 05.03.2019 14:30, 107565445 wrote: Hi Guys, I want to contribute to Apache Flink. Would you please give me the permission as a contributor? My JIRA ID is zhouhuiyu.

Re: Error on importing flink source code into Intellij

Usually when I run into this i use "File -> Invalidate Caches / Restart... -> Invalidate and restart" On 05.03.2019 16:15, Felipe Gutierrez wrote: Hello, I imported the flink source code at my Intellij IDE following the steps described here https://ci.apache.org/projects/flink/flink-docs-stabl

Re: [DISCUSS] Improve the flinkbot

The image didn't go through. I would keep it as is; imo there are significantly more important things that I'd like Robert to spend time on. (literally everything in the Feature requests section) If we want to better distinguish new PRs I would suggest to either a) introduce a dedicated "New

Re: [DISCUSS] Improve the flinkbot

Hey Kurt, thanks a lot for this idea. My reasoning behind using just one color is the following: I wanted to use one color per category of labels. So when we are introducing labels for components, that it'll look like this: [image: image.png] But we could of course also go with color families pe

Re: [DISCUSS] A more restrictive JIRA workflow

Hi Tison, I also thought about this. Making a person a "Contributor" is required for being an "Assignable User", so normal Jira accounts can't be assigned to a ticket. We could make every Jira user an "Assignable User", but restrict assigning a ticket to people with committer permissions. There ar

[jira] [Created] (FLINK-11837) Improve blink internal data foramt

Jingsong Lee created FLINK-11837: Summary: Improve blink internal data foramt Key: FLINK-11837 URL: https://issues.apache.org/jira/browse/FLINK-11837 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-11836) Update NOTICE-binary and licenses-binary for Flink 1.8.0

Aljoscha Krettek created FLINK-11836: Summary: Update NOTICE-binary and licenses-binary for Flink 1.8.0 Key: FLINK-11836 URL: https://issues.apache.org/jira/browse/FLINK-11836 Project: Flink

Re: [DISCUSS] A more restrictive JIRA workflow

Hi devs, Just now I find that one not a contributor can file issue and participant discussion. One becomes contributor can additionally assign an issue to a person and modify fields of any issues. For a more restrictive JIRA workflow, maybe we achieve it by making it a bit more restrictive granti

Re: Apply for rights for contributor

Resolved in a duplicate thread sent 1 hour before this one. On 06.03.2019 10:42, zoudan wrote: Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is FrankZou.

Re: Apply for rights for contributor

You now have contributor permissions. On 06.03.2019 09:49, zoudan wrote: Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is FrankZou.

Re: Apply for rights of contributor

You now have contributor permissions. On 06.03.2019 09:42, zoudan wrote: Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is Zou Dan.

Apply for rights for contributor

Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is FrankZou.

Apply for rights for contributor

Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is FrankZou.

Apply for rights of contributor

Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is Zou Dan.

[jira] [Created] (FLINK-11835) ZooKeeperLeaderElectionITCase#testJobExecutionOnClusterWithLeaderChange failed

Gary Yao created FLINK-11835: Summary: ZooKeeperLeaderElectionITCase#testJobExecutionOnClusterWithLeaderChange failed Key: FLINK-11835 URL: https://issues.apache.org/jira/browse/FLINK-11835 Project: Flink

[jira] [Created] (FLINK-11834) Introduce flink logical relational nodes

godfrey he created FLINK-11834: -- Summary: Introduce flink logical relational nodes Key: FLINK-11834 URL: https://issues.apache.org/jira/browse/FLINK-11834 Project: Flink Issue Type: New Feature