Re: Task Manager detached under load

Hi Vishal, you should not need to configure anything else. Cheers, Till On Sat, Jun 30, 2018 at 7:23 PM Vishal Santoshi wrote: > A clarification.. In 1.5 with custom heartbeats are there additional > configurations we should be concerned about ? > > On Fri, May 25, 2018 at 10:17 AM, Steven Wu

Re: Using Google Cloud Storage for checkpointing

Glad to hear it! On Sat, Jun 30, 2018 at 9:48 AM Rohil Surana wrote: > Thanks Till, > > On trying locally the setup seemed to have worked until I submitted jobs > to it, it started giving me ClassNotFoundExceptions for some other classes. > So I created a new shaded fat jar from the gcs-connecto

Re: Task Manager detached under load

A clarification.. In 1.5 with custom heartbeats are there additional configurations we should be concerned about ? On Fri, May 25, 2018 at 10:17 AM, Steven Wu wrote: > Till, thanks for the follow-up. looking forward to 1.5 :) > > On Fri, May 25, 2018 at 2:11 AM, Till Rohrmann > wrote: > >> Hi S

Re: Displaying topic data with Flink streaming

Hi Mich, How did you setup your local Kafka cluster, did you produce any message to it? Seems like you are using a standard local Kafka cluster setup for testing: "bootstrap.servers", "localhost:9092" "zookeeper.connect", "localhost:2181" so probably you need to manually produce some data, probab

Re: error: object connectors is not a member of package org.apache.flink.streaming

Thanks Rong This worked. $FLINK_HOME/bin/start-scala-shell.sh local --addclasspath /home/hduser/jars/flink-connector-kafka-0.9_2.11-1.5.0.jar:/home/hduser/jars/flink-connector-kafka-base_2.11-1.5.0.jar Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEW

Re: How to use broadcast variables in data stream

Hi Zhen, This might be a rather inefficient solution. We have encountered situations that we need to have some daily config update pushed to our flink streaming application, where the state is very large (but keyed). We end-up having a service to push that data into a separated kafka stream (which

Re: error: object connectors is not a member of package org.apache.flink.streaming

Hi Mich, Ted is correct, Flink release binary does not include any connectors and you will have to include the appropriate connector version. This is to avoid dependency conflicts between different Kafka releases. You probably need the specific Kafka connector version jar file as well, so in your

Re: Displaying topic data with Flink streaming

Please try new FlinkKafkaConsumer09[String]("md", new SimpleStringSchema(), properties).setStartFromEarliest() and try again. Cheers Minglei. > 在 2018年6月30日，下午10:08，Mich Talebzadeh 写道： > > > Hi, > > I have a streaming topic called "md" that displays test market data. > > I have written a s

Re: The difference between legacy mode and new mode

From my point of view, you can choose one of each. But prefer the new mode. And in the future, there is a plan to remove the legacy mode. Please see more about new mode [1] Cheers Minglei [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077

The difference between legacy mode and new mode

Hi all Could you please describe the difference between legacy mode and new mode? I find the configuration here: https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#legacy. I don't know which one I should choose. Thanks.

Displaying topic data with Flink streaming

Hi, I have a streaming topic called "md" that displays test market data. I have written a simple program to stream data in via kafka into flinl. Flink version 1.5 Kafka version 2.12 This is the sample program in scala that compiles ok in start-scala-shell.sh import java.util.Properties import

Re: How to use broadcast variables in data stream

Hi，Fabian： I use connected stream to solve this problem，one is config stream that load some config data from redis.another is real data stream. Due to the config data is vary big, then the config stream is slowly than the real data stream. When use some config to deal data in flatmap2, it ar

Re: error: object connectors is not a member of package org.apache.flink.streaming

Please add flink-connector-kafka-base_2.11 jar to the classpath. On Sat, Jun 30, 2018 at 1:06 AM, Mich Talebzadeh wrote: > > Great Ted added that jar file to the classpath > > Running this code > > import org.apache.flink.streaming.api.scala._ > import org.apache.flink.streaming.connectors.kafka

Re: Using Google Cloud Storage for checkpointing

Thanks Till, On trying locally the setup seemed to have worked until I submitted jobs to it, it started giving me ClassNotFoundExceptions for some other classes. So I created a new shaded fat jar from the gcs-connector source code, which worked locally and on K8s. Thanks for all the help. Rohil