Re: Does Flink operators synchronize states?

parallel operations), for which they may also need to revise their scheduling model. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/flink-architecture.html On Fri, Oct 30, 2020 at 10:05 AM Yuta Morisawa mailto:yu-moris...@kddi-research.jp>> wrote: Hello,

Does Flink operators synchronize states?

Hello, I am wondering whether Flink operators synchronize their execution states like Apache Spark. In Apache Spark, the master decides everything, for example, it schedules jobs and assigns tasks to Executors so that each job is executed in a synchronized way. But Flink looks different. It a

Re: How to use Lo-level Joins API

essed element. Best, Victor On 2019/8/7, 10:56 AM, "Yuta Morisawa" wrote: Hi Yun Thank you for replying. >Have you set a default value for the state ? Actually, the constructor of the ValueStateDescriptor with default value is deprecated

Re: How to use Lo-level Joins API

Hi Yun Thank you for replying. >Have you set a default value for the state ? Actually, the constructor of the ValueStateDescriptor with default value is deprecated so I don't set it. The problem occurs when the stream1 comes first. I made sure the 'ValueState data' has data from stream

How to use Lo-level Joins API

Hi I am trying to use low-level joins. According to the doc, the way is creating a state and access it from both streams, but I can't. https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/process_function.html This is a snippet of my code. It seems that the process

Re: How to access state from another map functions

different machines. On Mon, Jul 29, 2019 at 11:12 AM Yuta Morisawa mailto:yu-moris...@kddi-research.jp>> wrote: Hi all I want to use the same state for multiple times. stream .keyBy(0) .map(new MyRichMapfunction) ... .keyBy(0) .map(new MyRichMapfunction

How to access state from another map functions

Hi all I want to use the same state for multiple times. stream .keyBy(0) .map(new MyRichMapfunction) ... .keyBy(0) .map(new MyRichMapfunction2) In this snippet, I want access the same state in MyRichMapfunction and MyRichMapfunction2, but I failed in spite of keying the same key. Is there any

Re: How to update Flink applications gracefully

Motavalli *From:* Yuta Morisawa *Sent:* Tuesday, December 18, 2018 6:30:09 AM *To:* user@flink.apache.org *Subject:* How to update Flink applications gracefully Hi all Now I'm trying to update my streaming application.

How to update Flink applications gracefully

Hi all Now I'm trying to update my streaming application. But I have no idea how to update it gracefully. Should I stop it, replace a jar file then restart it? In my understanding, in that case, all the state will be recovered if I use checkpoints in a persistent storage. Is this correct? Tha

Re: Some question about document

in flink's documentation is in terms of the types flink supported. But for all the other arbitrary types kryo is the first choice. 2018-07-12 9:55 GMT+08:00 Yuta Morisawa <mailto:yu-moris...@kddi-research.jp>>: Thank you for your answer. > For POJOs Flink has a custo

Re: Some question about document

se Avro as a fallback. On 11.07.2018 09:24, Yuta Morisawa wrote: Hi all Now, I'm reading Flink document and I have some points to feel difficult to get an idea. I'd appreciate if you tell it me. 1,TypeInformation  I understand TypeInformation is used for selecting relevant serializer

Some question about document

Hi all Now, I'm reading Flink document and I have some points to feel difficult to get an idea. I'd appreciate if you tell it me. 1,TypeInformation I understand TypeInformation is used for selecting relevant serializer and comparator. But, the ducument doesn't specify if it has another way

Re: How to get kafka partition ID?

at 12:11 AM, Yuta Morisawa mailto:yu-moris...@kddi-research.jp>> wrote: Hi I want to apply a window function simply on data from each kafka-partition(I mean I don't need any grouping and just want to process data parallely). I think the best way to do so is drawing

How to get kafka partition ID?

Hi I want to apply a window function simply on data from each kafka-partition(I mean I don't need any grouping and just want to process data parallely). I think the best way to do so is drawing kafka partition id and use it for keyBy function. For example, val data = stream.keyBy(kafka-parti

Re: CPU Cores of JobManager

ght that be an issue? On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa mailto:yu-moris...@kddi-research.jp>> wrote: > Which version of Fllink are you using? I'm using Flink 1.3.0. > Is the high CPU usage caused by the browser or the JVM? The CPU usage of browse

Re: CPU Cores of JobManager

JobManager, but I don't know of a good way to do that without modifying the source. On 06.12.2017 01:20, Yuta Morisawa wrote: Hi Timo I execute streaming job without checkpointing and I don't configure any state backend, so it may be "MemoryStateBackend". Actually, my stre

Re: CPU Cores of JobManager

o much CPU resources? Regards, Timo Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: Hi Now I am looking for the way to increase the number of allocated CPU cores because my JobManagaer WEBUI is very heavy and sometimes freeze. I think this is caused by the resource shortage of JobManager. H

Re: CPU Cores of JobManager

ources? Regards, Timo Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa: Hi Now I am looking for the way to increase the number of allocated CPU cores because my JobManagaer WEBUI is very heavy and sometimes freeze. I think this is caused by the resource shortage of JobManager. How can I increase the

CPU Cores of JobManager

Hi Now I am looking for the way to increase the number of allocated CPU cores because my JobManagaer WEBUI is very heavy and sometimes freeze. I think this is caused by the resource shortage of JobManager. How can I increase the number of CPU for JobManager in YARN mode? Thanks Yuta --

Re: Streaming API has a long delay at the beginning of the process.

re never GC'd (unless a job was cancelled), so the setup cost is a one time thing that only happens when the job is started. Best, Fabian 2017-09-15 12:43 GMT+02:00 Yuta Morisawa <mailto:yu-moris...@kddi-research.jp>>: Hi Fabian, Thank you for your description. This is

Re: Streaming API has a long delay at the beginning of the process.

Kafka before subsequent tasks have been started. So, there is quite a lot happening when you call execute() including network communication and task deployment. Hope this helps, Fabian 2017-09-15 4:25 GMT+02:00 Yuta Morisawa <mailto:yu-moris...@kddi-research.jp>>: Hi, Fabian >

Re: Streaming API has a long delay at the beginning of the process.

ycle of tasks. Best, Fabian 2017-09-12 11:02 GMT+02:00 Yuta Morisawa <mailto:yu-moris...@kddi-research.jp>>: Hi, I am worrying about the delay of the Streaming API. My application is that it gets data from kafka-connectors and process them, then push data to kafka-pro

Streaming API has a long delay at the beginning of the process.

Hi, I am worrying about the delay of the Streaming API. My application is that it gets data from kafka-connectors and process them, then push data to kafka-producers. The problem is that the app suffers a long delay when the first data come in the cluster. It takes about 1000ms to process data