Questions for platform to choose

Hello, We have all of spark, flink, storm, kafka installed. For realtime streaming calculation, which one is the best above? Like other big players, the logs in our stack are huge. Thanks.

How to shorten MATCH_RECOGNIZE's DEFINE clause

Hi, Flink relational apis with MATCH_RECOGNITION looks very attractive and promising but it's currently not easy to use at all. While looking into [1], especially the following DEFINE clause, > DEFINE > PRICE_DOWN AS > (LAST(PRICE_DOWN.price, 1) IS NULL AND PRICE_DOWN.price < START_ROW.price) OR

Re: What is the recommended way to run flink with high availability on AWS?

Thanks Kali for the information. However, it doesn't work for me, because I need features in Flink 1.7.x or later and use manged Amazon MSK. -- Pei On Tue, Aug 20, 2019 at 7:17 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > check this out AWS managed Flink cluster in aws kin

Not seeing Kafka partition- and topic-level metrics

Hello folks, My question is about Kafka client metrics exported by Flink. According to the Flink documentation : "The producers export Kafka’s internal metrics through Flink’s metric system

Re: build Flink master brach fail due to npm

HI, Could you try adding "-Dskip.npm" when building the project? This is just a work-around way to skip this installation. Though I don't have much experience of npm. Maybe someone else could help. Thanks, Biao /'bɪ.aʊ/ On Tue, 20 Aug 2019 at 16:23, 刘波 <1091643...@qq.com> wrote: > Hi, users!

What is the recommended way to run flink with high availability on AWS?

Hi there, Looking at Flink document [1], EMR is the recommended way to run Flink on AWS. However, Flink is currently not supported in an EMR cluster with multiple master nodes [2]. For example, I can create a EMR HA cluster with Zookeeper, Hive, etc. But, if I add Flink to the application list whe

Error while sinking results to Cassandra using Flink Cassandra Connector

Hi, I am facing the issue of writing the data stream result into Cassandra. I tried with a sample job of just sinking a Tuple into Cassandra. And I keep on getting “java.lang.NoSuchMethodError” while running the job on Flink deployed on Kubernetes. Here is the sample WriteToCassandraJob: publ

Re: Configuring logback

Hi How to build a Graph using Flink Gelly from a text file that consists of edge list. regards TR RAO From: "Vishwas Siravara" To: "user" Sent: Tuesday, August 20, 2019 9:17:01 PM Subject: Configuring logback Hi guys, I am using logback for my application logs. I have logback.xml a

Configuring logback

Hi guys, I am using logback for my application logs. I have logback.xml as a part of my fat jar that I submit to flink via command line flink run "...". When I run my application from IDE , the appenders are what I have set in my logback but when I run from command line the appender defaults to the

Re: Configuring logback for my flink job

My logback is a part of my application jar file, do you mean I should externalize it and pass set its location in FLINK_CONF_DIR ? like export FLINK_CONF_DIR=/home/was/ where I have logback.xml in /home/was/ ? Thanks, Vishwas On Mon, Aug 19, 2019 at 10:31 PM Yang Wang wrote: > Hi Vishwas, > >

combineGroup get false results

Hi, I used the combineGroup function to reduce groups of a very large dataset. By modifying the parallelism to 1 I have a different results with a parallelism to 8, Knowing that the good results are those obtained with the parallelism with 1. I also used table api to group dataset and select sum

Re: Stale watermark due to unconsumed Kafka partitions

OK that makes sense, I can see how sharing the same Assigner instance between partitions could be tricky, specially if you're handling state. Good to hear this is not expected behaviour, I'm using Flink's JDBCAppendTableSink so nothing particularly unconventional (and I get the same result with JD

build Flink master brach fail due to npm

Hi, users! When I'm build the Flink master branch from github, it's failed due to the flink-runtime-web module. nodejs v10.15.2 npm6.10.3 Javaopenjdk version "11.0.4" 2019-07-16 maven 3.6.0 anyone got the same problem? any suggestions to resolve it... thanks