Re: Temporary failure in name resolution

HI , i checked the code again the figure out where the problem can be i just wondered if im implementing the Evictor correctly ? full code https://gist.github.com/miko-code/6d7010505c3cb95be122364b29057237 public static class EsbTraceEvictor implements Evictor { org.slf4j.Logger LOG =

Fwd: Updating Broadcast Variables

Hi Guys, Quick question about broadcast variables. How would one periodically update the value of a broadcast variable? Suppose my broadcast variable is used for lookups, and these lookup values can change over time (i.e. some of the lookup values get removed, whilst others get added). Is it pos

Re: Fw: 1.4.2 - Unable to start YARN containers with more than 1 vCores per Task Manager

Thanks Shashank, I will check ot out. -- Ashish On Tue, Apr 3, 2018 at 10:11 AM, shashank734 wrote: CHeck in your Yarn configuration, Are you using DeafaultResourceCalculater which only uses memory while allocating resources. So you have to change it to DominantResourceCalculator.     ya

Re: bad data output

Thanks, As of now I have decided to write it to hdfs from within the function. Thanks On Tue, Apr 3, 2018 at 10:58 AM, Kostas Kloudas wrote: > Hi Darshan, > > You can use side outputs [1] and a process function to split the data in > as many streams as you want, > e.g. correct, fixable and wro

SideOutput Issue

Hi All, I'm having issues with creating side outputs. There are two input sources (both from kafka) and they are connected and fed into a co-process function. Inside the co-process, the regular data stream outputs a POJO and in processElement2 there is a periodic timer which creates the side outpu

Re: Watermark Question on Failed Process

Thanks Timo for your response and the references. I will try BoundedOutOfOrdernessTimestampExtractor, and if it does't work as expected, I will handle it as a separated pipeline. Also, is there a way to retrieve the last watermark before/after failure? So maybe I can persist the watermark to exter

Re: Multiple Async IO

Hi Maxim, If reducing latency is the goal, then option #1 seems better. Though you’d need additional logic inside of your AsyncFunction to run all 20 queries in parallel. I’d also consider a third option... Use a FlatMapFunction to create 20 copies of the event (assuming it’s not large), with

Multiple Async IO

Hi everyone, I'm writing streaming job which needs to query Cassandra for each event multiple times, around 20. I would like to use Async IO for that but not sure which option to choose: 1. Implement One AsyncFunction with 20 queries inside 2. Implement 20 AsyncFunctions, each with 1 query inside

Re: Multiple (non-consecutive) keyBy operators in a dataflow

I don’t think there are any particular implications. I would suggest to go for a simple keyBy and think about optimization if there should actually be a problem at hand. Best, Stefan > Am 03.04.2018 um 17:08 schrieb Timo Walther : > > @Richter: Are you aware of any per-key state size performan

Re: Multiple (non-consecutive) keyBy operators in a dataflow

@Richter: Are you aware of any per-key state size performance implications? Am 03.04.18 um 16:56 schrieb au.fp2018: Thanks Timo/LiYue, your responses were helpful. I was worried about the network shuffle with the second keyBy. The first keyBy is indeed evenly spreading the load across the node

Re: Reading data from Cassandra

Sorry, the examples contain only sinks. You could do an approach similar to the one mentioned here: https://stackoverflow.com/questions/42617575/read-write-data-into-cassandra-using-apache-flink-java-api Either put this into your custom SourceFunction or use Async IO [1] with a Cassandra Clien

Re: Multiple (non-consecutive) keyBy operators in a dataflow

Thanks Timo/LiYue, your responses were helpful. I was worried about the network shuffle with the second keyBy. The first keyBy is indeed evenly spreading the load across the nodes. As I mentioned my concern was around the amount of state in each key. Maybe I am trying to optimize pre-maturely here

Re: Reading data from Cassandra

Hi Soheil, yes Flink supports reading from Cassandra. You can find some examples here: https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example Regards, Timo Am 31.03.18 um 20:22 schrieb Soheil

Re: Fw: 1.4.2 - Unable to start YARN containers with more than 1 vCores per Task Manager

CHeck in your Yarn configuration, Are you using DeafaultResourceCalculater which only uses memory while allocating resources. So you have to change it to DominantResourceCalculator. yarn.scheduler.capacity.resource-calculator org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

[ANNOUNCE] Weekly community update #14

Dear community, here is the weekly community update thread #14. Please post any news and updates you want to share with the community to this thread. # Release 1.5 The community could cut down the number of release blockers significantly. A first release candidate has been created in order to fa

Re: Task Manager fault tolerance does not work

There is a JIRA issue for the problem: https://issues.apache.org/jira/browse/FLINK-9120. Mirroring my response to this thread: The logs (attached to the JIRA ticket) show that the JM did not yet recognize the killed TM as killed when trying to restart. Thus, it tries to re-deploy tasks to this mac

Re: Temporary failure in name resolution

Hi Timo, we do have similar issue, TM got killed by a job. Is there a way to monitor JVM status? If through the monitor metrics, what metric I should look after? We are running Flink on K8S. Is there a possibility that a job consumes too much network bandwidth, so JM and TM can not connect? On Tue

Re: Task Manager fault tolerance does not work

@Till: Do you have any advice for this issue? Am 03.04.18 um 11:54 schrieb dhirajpraj: What I have found is that the TM fault tolerance behaviour is not consistent. Sometimes it works and sometimes it doesnt. I am attaching my java code file (which is the main class). What I did was: 1) Run cl

Re: Kafka exceptions in Flink log file

Hi Alex, which version of Flink are you running? There were some class loading issues with Kafka recently. I would try it with the newest Flink version. Otherwise ClassNotFoundException usually indicates that something is wrong with your dependencies. Maybe you can share your pom.xml with us.

Re: Flink and Docker ?

Hi, we are using this docker on K8S + S3. https://github.com/docker-flink/docker-flink It works fine for us. On Tue, Apr 3, 2018 at 1:00 AM Christophe Salperwyck < christophe.salperw...@gmail.com> wrote: > Hi, > > I didn't try docker with Flink but I know that those guys did: > https://github.co

Re: Side outputs never getting consumed

Hi Julio, I tried to reproduce your problem locally but everything run correctly. Could you share a little example job with us? This worked for me: class TestingClass { var hello:Int =0 } class TestAextends TestingClass { var test:String = _ } def main(args: Array[String]) { // set u

Fw: 1.4.2 - Unable to start YARN containers with more than 1 vCores per Task Manager

Hi All,   I had been using the following command in a Lab environment successfully in 1.3 Flink version.   yarn-session.sh -n 4 -s 4 -jm 2048 -tm 2048 -Dyarn.containers.vcores=2 -nm infra.test3   As expected, I see 4 TMs with 16 slots and taking 8 vCores from YARN. In a new Prod

Re: Savepointing with Avro Schema change

Hi Aneesha, as far as I know Avro objects were serialized with Flink's POJO serializer in the past. This behavior changed in 1.4. @Gordon: do you have more information how good we support Avro schema evolution today? Regards, Timo Am 03.04.18 um 12:11 schrieb Aneesha Kaushal: Hello, I hav

Re: Watermark Question on Failed Process

Hi Chengzhi, if you emit a watermark even though there is still data with a lower timestamp, you generate "late data" that either needs to be processed in a separate branch of your pipeline (see sideOutputLateData() [1]) or should force your existing operators to update their previously emitte

Re: Multiple (non-consecutive) keyBy operators in a dataflow

Hi Andre, every keyBy is a shuffle over the network and thus introduces some overhead. Esp. serialization of records between operators if object reuse is disabled by default. If you think that not all slots (and thus all nodes) are not fully occupied evenly in the first keyBy operation (e.g.

Re: Temporary failure in name resolution

Hi Miki, for me this sounds like your job has a resource leak such that your memory fills up and the JVM of the TaskManager is killed at some point. How does your job look like? I see a WindowedStream.apply which might not be appropriate if you have big/frequent windows where the evaluation h

Savepointing with Avro Schema change

Hello, I have a flink job which processes a stream of Event (an avro object) and creates Sessions (another avro object) using Session-Windows. I am not able to recover my job from save point when I try to make some changes in the schema of Event object or in schema of Sessions. Is there anyon

Re: bad data output

Hi Darshan, You can use side outputs [1] and a process function to split the data in as many streams as you want, e.g. correct, fixable and wrong. Each side output will be a separate stream that your can process individually. You can always send the “bad data” directly from your process functio

Re: Task Manager fault tolerance does not work

What I have found is that the TM fault tolerance behaviour is not consistent. Sometimes it works and sometimes it doesnt. I am attaching my java code file (which is the main class). What I did was: 1) Run cluster with JM on machine A, one TM on machine B and one TM on machine C 2) Submit a job to

Re: Job restart hook

Hi Navneeth, If I understand correctly, you have a job with parallelism p=20, a TM goes down (eg. with 4 slots), and you want until the TM comes up, to run the job with p=16 and then re-running it with 20 again, when the TM comes up. If this is the case, one important thing to keep in mind is t

Re: Task Manager fault tolerance does not work

Could you provide a little reproducible example? Which file system are you using? This sounds like a bug to me that should be fixed if valid. Am 03.04.18 um 11:28 schrieb dhirajpraj: I have not specified any parallelism in the job code. So I guess, the parallelism should be set to parallelism.d

Re: subuquery about flink sql

Hi, there are multiple issues in your query. First of all, "SELECT DISTINCT(user), product" is MySQL specific syntax and is interpreted as "SELECT DISTINCT user, product" which is not what you want I guess. Secondly, SQL windows can only be applied on time attributes. Meaning: "As long as a

Re: Task Manager fault tolerance does not work

I have not specified any parallelism in the job code. So I guess, the parallelism should be set to parallelism.default defined in the flinkConfig.yaml. An update: The TMs were on different machines and I was using FsStateBackend with state backend directories pointing to instance specific file pa

Re: Task Manager fault tolerance does not work

Hi, does your job code declare a higher parallelism than 2? Or is submitted with a higher parallelism? What is the Web UI displaying? Regards, Timo Am 03.04.18 um 10:48 schrieb dhirajpraj: Hi, I have done that env.enableCheckpointing(5000L); env.setRestartStrategy(RestartStrategies.fixedDela

Re: Task Manager fault tolerance does not work

Hi, I have done that env.enableCheckpointing(5000L); env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, 5000)); -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: subuquery about flink sql

You should add a column TUMBLE_ROWTIME(t, INTERVAL '60' SECOND) AS t to the select part of your subquery. 韩宁宁 <453673...@qq.com> 于 2018年4月3日周二 下午3:34写道： > Thank you for your reply. > > I think the table registration no problem。I guess it's a question of > subquery。 > It's no problem to execut

Re: Task Manager fault tolerance does not work

Please make sure you have set a number of re-tries and have checkpointing activated if you use streaming. On Fri, Mar 30, 2018 at 1:59 PM, dhirajpraj wrote: > HI, > I have set up a flink 1.4 cluster with 1 job manager and two task managers. > The configs taskmanager.numberOfTaskSlots and paralle

Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes

Hi! This is pretty much all in Hadoop's magic, from Flink's view, once this has been delegated to s3a. I seem to recall that there was something with older hadoop-aws versions or AWS SDK versions. There were cases where that version needed to be bumped. What we use in the pre-bundled s3 connecto

Re: Flink Kafka producer: Object of class is not serializable

As answered on SO, the problem is the KeyedDeserializationSchema which is an anonymous inner class. On 31.03.2018 20:25, Soheil Pourbafrani wrote: I got an error in using Flink Kafka connector for producing data. I describe the problem here

Re: Flink and Docker ?

Hi, I didn't try docker with Flink but I know that those guys did: https://github.com/big-data-europe/docker-flink Perhaps it is worth having a look there. BR, Christophe 2018-04-03 9:29 GMT+02:00 Esa Heikkinen : > Hi > > > > I have noticed that Flink can be pretty tedious to install and build

?????? subuquery about flink sql

Thank you for your reply. I think the table registration no problem??I guess it's a question of subquery?? It's no problem to execute this SQL?? select user, count(product), TUMBLE_START(t, INTERVAL '60' SECOND) as wStart, TUMBLE_END(t, INTERVAL '60' SECOND) as wE

Flink and Docker ?

Hi I have noticed that Flink can be pretty tedious to install and build first applications from scratch. Especially if the application is little bit complex. There are also little bit different development and run time environments, which require different software components with correct versi