Providing files while application mode deployment

Hi all. While running Flink jobs in application mode on YARN and Kuber, we need to provide some configuration files to main class. Is there any option on Flink CLI to copy local files on cluster without manually copying on DFS or in docker image, something like *--files* option in spark-submit?

Reuse in Blink execution plan

Hi all. Using SQL with blink planner for batch calculations, i see *Reused* nodes in Optimized Execution Plan while making self join operations: == Optimized Execution Plan == Union(all=[true], union=[id, v, v0, w0$o0]) :- OverAggregate(orderBy=[id DESC], window#0=[ROW_NUMBER(*) AS w0$o0 ROWS B

Re: State expiration in Flink

----------- > *From:* Vasily Melnik > *Sent:* Saturday, May 30, 2020 23:29 > *To:* user > *Subject:* State expiration in Flink > > Hi . > I'm a bit confused with this point in State TTL documentation: > " By default, expired values are explicitly removed

State expiration in Flink

Hi . I'm a bit confused with this point in State TTL documentation: " By default, expired values are explicitly removed on read, such as ValueState#value, and periodically garbage collected in the background if supported by the configured state backend. " Does it mean, that if i have only one even

Re: REST rescale with Flink on YARN

e operation. > > On 23/01/2020 13:23, Vasily Melnik wrote: > > Hi all. > I'm using Flink 1.8 on YARN with CDH 5.12 > When i try to perform rescale request: > > curl -v -X PATCH > '/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescali

REST rescale with Flink on YARN

Hi all. I'm using Flink 1.8 on YARN with CDH 5.12 When i try to perform rescale request: curl -v -X PATCH '/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3

Re: State name uniqueness

Gordon, great thanks! С уважением, Василий Мельник On Mon, 20 Jan 2020 at 13:56, Tzu-Li (Gordon) Tai wrote: > Hi Vasily, > > State names need to be unique within operators only. > > Cheers, > Gordon > > On Mon, Jan 20, 2020 at 10:58 AM Vasily Melnik < > vasily

State name uniqueness

Hi all, I'm a bit confused with state name uniqueness. Should it be unique within operator only, or within entire job? С уважением, Василий Мельник

Re: Broadcast checkpoint serialization fail

ttp://www.gbconsulting.ru/> === Моб. тел.: +7 (903) 101-43-71 vasily.mel...@glowbyteconsulting.com On Fri, 15 Nov 2019 at 14:29, Vasily Melnik < vasily.mel...@glowbyteconsulting.com> wrote: > Hi all. > In Flink 1.8 we have strange exception that causes job failing: > > 2019

Broadcast checkpoint serialization fail

Hi all. In Flink 1.8 we have strange exception that causes job failing: 2019-11-14 15:52:52,071 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- op4 (1/1) (797d4c2b85010dab6be5e1d06ff6493a) switched from RUNNING to FAILED. AsynchronousException{java.lang.Exception: Could not m

Re: Initialization of broadcast state before processing main stream

gt; >> >>- you can change your storage about dimension table, e.g. Redis or >>MySQL and so on to do the stream and dimension table join; >>- you can inject some control event in your broadcast stream to mark >>the stream is end and let the fact stream wait

Initialization of broadcast state before processing main stream

Hi all. In our task we have two Kafka topics: - one with fact stream (web traffic) - one with dimension We would like to put dimension data into broadcast state and lookup on int with facts. But we see that not all dimension records are put into state before first fact record is processed, so loo

Re: Set flink-conf parameters dynamicly

Thanks, Dian! It works for me) С уважением, Василий Мельник On Thu, 19 Sep 2019 at 18:10, Dian Fu wrote: > Hi Vasily Melnik, > > If you use yarn mode, you could try to use the option "-yD value>" during submitting the job [1] to change the configuration > d

Set flink-conf parameters dynamicly

Hi, all I wonder is it possible to pass custom flink-conf file as run util parameter or change config parameters in Java code manually? I'd like to change metrics.scope parameter for each job independently. С уважением, Василий Мельник

Re: Flink metrics scope for YARN single job

. I would like to > discuss with the devs to hear their opinions. Will give you a feedback if > there is a conclusion. > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Wed, 14 Aug 2019 at 19:46, Vasily Melnik < > vasily.mel...@glowbyteconsulting.com> wrote: > >>

Flink metrics scope for YARN single job

Hi, I want to run Flink apps on YARN in single job mode and keep metrics in Graphite. But as i see, the only variable i can use for JobManager scope customization is : https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables Is there any way to distingui

Re: Exactly-once semantics in Flink Kafka Producer

wrote: > Hi Vasily, > > as far as I know, by default console-consumer reads uncommited. > Try setting isolation.level to read_committed in console-consumer > properties. > > Hope this helps, > Maxim. > > On Fri, Aug 2, 2019 at 12:42 PM Vasily Melnik < >

Re: Exactly-once semantics in Flink Kafka Producer

u're running in a proper > cluster environment. Flink is able to recover using checkpoints when only > part of the cluster fails, not when the whole job is stopped. For full > stops you need to specify the checkpoint manually. > > Hope that helps! > > > On Fri, 2 Aug 2019

Exactly-once semantics in Flink Kafka Producer

I,m trying to test Flink 1.8.0 exactly-once semantics with Kafka Source and Sink: 1. Run flink app, simply transferring messages from one topic to another with parallelism=1, checkpoint interval 20 seconds 2. Generate messages with incrementing integer numbers using Python script each