Classloader error after SSL setup

Background: We have a setup of Flink 1.3.1 along with a secure MAPR cluster (Flink is running on mapr client nodes). We run this flink cluster via flink-jobmanager.sh foreground and flink-taskmanager.sh foreground command via Marathon. In order for us to make this work, we had to add -Djavax.net

[no subject]

-- Yours Sincerely, Aniket S Deshpande.

Re: Custom Metrics Reporting in Flink

This doesn't sound like a proper use-case for the metric system, but for side-outputs with a dedicated sink. My reasoning is that it sounds like you want to push data, whereas the metrics work by drawing a snapshot of all metrics and writing them out, i.e. one value per metric. On 03.10.2017

Re: At end of complex parallel flow, how to force end step with parallel=1?

Gábor ​, Thank you for the reply, I gave that a go and the flow still showed parallel 90 for each step. Is the ui not 100% accurate perhaps? To get around it for now I implemented a partitioner that threw all the data to the same partition, hack but works!​ On Tue, Oct 3, 2017 at 4:12 AM, Gábor

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated for S3 access

I am using S3 for checkpointing and external ckp as well. s3a://bucket/checkpoints/e58d369f5a181842768610b5ab6a500b I have this exception, and not sure what I can do with it. I guess to configure hadoop to use some SSLFactory? I am not using hadoop, I am on kubernetes (in AWS) with S3 Thanks!

Re: Stream Task seems to be blocked after checkpoint timeout

Sure, I opened Jira FLINK-7757 and this PR: https://github.com/apache/flink/pull/4764 . Best, Stefan > Am 03.10.2017 um 10:25 schrieb Tony Wei : > > Hi Stefan, > > Thank you very much. I will try to investigate what's the problem on my > cluster an

Custom Metrics Reporting in Flink

Hi, Is there any good example for custom metrics reporting in Flink? I tried to follow the documentation but failed to achieve the result. Basically my task is to identify the corrupt records based on a missing field and report the same via statsd reporter. Rahul Raj

Fw: Question on Flink on Window

From: Tay Zhen Shen Sent: Tuesday, 3 October, 2017 1:03 PM To: user@flink.apache.org Subject: Question on Flink on Window Hi, I'm currently trying to setup Flink 1.3.2 on my Windows 10. When i was running the start-local.bat file, the command prompt shows th

Re: Sink buffering

Anyone? :) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: kafka consumer parallelism

(Accidentally sent this to Timo instead of to-list...) Hi, What Timo says is true, but in case you have a higher parallism than the number of partitions (because you want to make use of it in a future operator), you could do a .rebalance() (see https://ci.apache.org/projects/flink/flink-docs-r

Re: Stream Task seems to be blocked after checkpoint timeout

Hi Stefan, Thank you very much. I will try to investigate what's the problem on my cluster and S3. BTW, Is there any Jira issue associated with your improvement, so that I can track it? Best Regards, Tony Wei 2017-10-03 16:01 GMT+08:00 Stefan Richter : > Hi, > > from the stack trace, it seems t

Re: At end of complex parallel flow, how to force end step with parallel=1?

Hi Garrett, You can call .setParallelism(1) on just this operator: ds.reduceGroup(new GroupReduceFunction...).setParallelism(1) Best, Gabor On Mon, Oct 2, 2017 at 3:46 PM, Garrett Barton wrote: > I have a complex alg implemented using the DataSet api and by default it > runs with parallel 90

Re: Stream Task seems to be blocked after checkpoint timeout

Hi, from the stack trace, it seems to me like closing the checkpoint output stream to S3 is the culprit: "pool-55-thread-7" #458829 prio=5 os_prio=0 tid=0x7fda180c4000 nid=0x55a2 waiting on condition [0x7fda092d7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsaf

RE: kafka consumer parallelism

Hi Robert, I had similar issue. For me the problem was that the topic was auto created with one partition. You can alter it to have 5 partitions using kafka-topics command. Example: kafka-topics --alter --partitions 5 --topic fix --zookeeper localhost:2181 Regards, Tovi -Original Message-