Question about Flink's savepoint

Hi all, I have some questions about the experience I had with the save point. So, last night I found my flink cluster's memory usage seemed wired, so I decided to 1. create a savepoint for the running job(there was only one job running at the time) 2. and then cancel the job from web UI 3. and re

Re: How to set jobmanager.rpc.address in TaskManger node in HA cluster

Sorry. I didn't put high-availability: zookeeper in taskmangers' flink-config.yml. After I fixed this, everything went well. On Fri, Jul 7, 2017 at 11:08 AM, Mu Kong wrote: > Hi all, > > I'm trying setup an HA Flink cluster with 3 job managers and 3 task > manage

How to set jobmanager.rpc.address in TaskManger node in HA cluster

Hi all, I'm trying setup an HA Flink cluster with 3 job managers and 3 task managers. After executing start-cluster.sh, 3 job managers are normally started. However, the task managers are still down due to an exception showed below: org.apache.flink.util.ConfigurationException: Config parameter '

Re: [QUESTION] OutOfMemoryError when writing into HDFS

t :) Best regards, Mu On Mon, Jun 12, 2017 at 10:45 AM, Ted Yu wrote: > Can you see if the following post helps in troubleshooting ? > > https://blog.fastthread.io/2016/07/06/troubleshoot- > outofmemoryerror-unable-to-create-new-native-thread/ > > Thanks > > On Sun, Jun 1

[QUESTION] OutOfMemoryError when writing into HDFS

Hi all, Thanks a lot for your work for the community! This question is more like a discuss. Currently, I'm experiencing an *OutOfMemoryError* when writing files from Kafka into HDFS using *BucketingSink*. The log looks like this: 2017-06-10 08:58:42,248 INFO org.apache.flink.runtime.taskmanager

Re: Got exception: unable to create new native thread when use BucketingSink writing to HDFS

Hi Till Rohrmann, Thanks for the prompt response! Limiting the number of open connections to the HDFS would be great. I created an issue here in JIRA: https://issues.apache.org/jira/browse/FLINK-6873 Best regards, Mu On Thu, Jun 8, 2017 at 7:06 PM, Till Rohrmann wrote: > Hi Mu Kong, &g

[jira] [Created] (FLINK-6873) Limit the number of open writers in file system connector

Mu Kong created FLINK-6873: -- Summary: Limit the number of open writers in file system connector Key: FLINK-6873 URL: https://issues.apache.org/jira/browse/FLINK-6873 Project: Flink Issue Type

Got exception: unable to create new native thread when use BucketingSink writing to HDFS

Hi all, I am new to flink. I tried to start a cluster over 3 servers, and the process speed was GREAT. However, after several hours' streaming, I got this error: *java.lang.OutOfMemoryError: unable to create new native thread* From: at org.apache.flink.streaming.connectors.fs.StreamWriterBase