Re: Flink 1.5.4 -- issues w/ TaskManager connecting to ResourceManager

We started to see same errors after upgrading to flink 1.6.0 from 1.4.2. We have one JM and 5 TM on kubernetes. JM is running on HA mode. Taskmanagers sometimes are loosing connection to JM and having following error like you have. *2018-09-19 12:36:40,687 INFO org.apache.flink.runtime.taskexecut

streaming state

tical experience though. Thanks *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>*

Re: streaming state

Thank you Stephan for the information! On Mon, Dec 14, 2015 at 5:20 AM Stephan Ewen wrote: > Hi Alex! > > Right now, Flink would not reuse Kafka's partitioning for joins, but > shuffle/partition data by itself. Flink is very fast at shuffling and adds > very little latency

kafka integration issue

quot;); properties.setProperty("group.id", "test"); val stream = env .addSource(new FlinkKafkaConsumer082[String]("topic", new SimpleStringSchema(), properties)) val counts = stream.map(f=>f.split(",")) print(counts) env.execute() } -- *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>*

Re: kafka integration issue

org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) at java.lang.Thread.run(Thread.java:745) On Tue, Jan 5, 2016 at 3:54 AM Till Rohrmann wrote: > Hi Alex, > > this is a bug in the `0.10` release. Is it possible for you to switch to > version `1.0-SNAPSHOT`. With this version, the error shou

Re: kafka integration issue

The version in "~/git/flink/" must be slightly outdated. Can you pull the > latest update of the 1.0-SNAPSHOT master and rebuild the code? > > Stephan > > On Tue, Jan 5, 2016 at 9:48 PM, Robert Metzger > wrote: > >> Hi Alex, >> >> How recent is your

sources not available for flink-streaming

*IntelliJ reports the following:* *Cannot download sources* Sources not found for: org.apache.flink:flink-streaming-scala:1.0-SNAPSHOT Would it be possible to publish sources for this artifact? -- *Alex Rovner* *Director, Data Engineering * *o:* 646.759.0052 * <http://www.magnetic.com/>*

"stepless" sliding windows?

Hey folks! I have an application that wants to use "stepless" sliding windows, i.e. we produce aggregates on every event. The windows need to be of a fixed size, but to have their start and end times update continuously, and I'd like to trigger on every event. Is this a bad idea? I've googled and

Re: "stepless" sliding windows?

} override fun getWindowSerializer(executionConfig: ExecutionConfig?): TypeSerializer { return TimeWindow.Serializer() } override fun isEventTime(): Boolean { return true } } On Tue, Oct 20, 2020 at 9:13 AM Alex Cruise wrote: > Hey folks! > >

Re: "stepless" sliding windows?

AM Danny Chan wrote: > >> The SLIDING window always triggers as of each step, what do you mean by >> "stepless" ? >> >> Alex Cruise 于2020年10月21日周三 上午1:52写道： >> >>> whoops.. as usual, posting led me to find some answers myself. Does this >>&g

Multiple side outputs of same type?

Hey folks, I have a program that demultiplexes input records from a shared prefix stream onto some number of suffix streams, which are allocated on boot based on configuration. At the moment I'm just duplicating the input records, and filtering out the wrong records in each suffix stream, but it'

Re: Multiple side outputs of same type?

uh... OK, thanks! 😅 -0xe1a On Fri, Dec 18, 2020 at 11:20 AM Arvid Heise wrote: > Hi Alex, > > not entirely sure how you reached your conclusion but afaik side output is > dispatched through the output tag. > > There are even tests in the code base [1] that use multiple out

Native Kubernetes deployment with GitOps

using the Flink CLI? In other words, I'd only use the Flink CLI to bootstrap the Git repository. Besides ease of setup, are there any other benefits to the native Kubernetes integration compared to standalone deployments? Thanks! Alex

Re: Native Kubernetes deployment with GitOps

deployment options on Kubernetes. As a side note, we're planning on using Flink in Application Mode. Alex On Jan 7, 2021, at 4:17 PM, Israel Ekpo mailto:israele...@gmail.com>> wrote: If you are just getting started, running it on Kubernetes could simplify that logistics and resources

Async + Broadcast?

Hi folks, I have a somewhat complex Flink job that has a few async stages, and a few stateful stages. It currently loads its configuration on startup, and doesn't attempt to refresh it. Now I'm working on dynamic reconfiguration. I've written a polling source which sends a configuration snapshot

Re: Async + Broadcast?

Thanks Austin! I'll proceed with my idea, and keep the bootstrap config :) -0xe1a On Wed, Apr 7, 2021 at 2:18 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Alex, > > I'm not sure if there is a best practice here, but what I can tell you is > tha

Re: Async + Broadcast?

lidateAndDistribute(initialConfigs)) val enriched: DataStream = AsyncDataStream.unorderedWait( validated, Enrich(), 5L, TimeUnit.SECONDS ) On Wed, Apr 7, 2021 at 11:28 PM Arvid Heise wrote: > Hi Alex, > > your approach is completely valid. What you want to achieve is that you

Possible way to avoid unnecessary serialization calls.

Dear entity that represents Flink user community, In order to formulate the question itself, I would need to describe the problem in many details, hence please bear with me for a while. I have following execution graph: KafkaSource -> Message parser -> keyBy -> TCP assembly -> keyBy -> Storage -

After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

Dear flink community, First I need provide some minimum information about my deployment scenario: I'm running application inside of Flink docker, below original Dockerfile: --- FROM flink:1.13.

Re: After upgrade from 1.11.2 to 1.13.0 parameter taskmanager.numberOfTaskSlots set to 1.

sider the possibility > of users mounting the configuration directly, and instead assumed that > modifications to the config always go through the FLINK_PROPERTIES > environment variable. > > That would also be the workaround for your issue. > > On 5/12/2021 2:06 PM, Alex Drobins

flink run jar throw java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

this problem? Can @Jark Wu give me some advice? Or can someone give me some advice? Thank you. 1. pom.xml > mysql > mysql-connector-java > 5.1.38 > 2. mvn clean; mvn package $ ll -rth target > > [±master ●] > total 32312 > drwxr-xr-x 3 alex staff96B Oct 30 11

Re: flink run jar throw java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

ve this problem. I will give you feedback if I have an answer. Thanks again, Yang and Frank vino yang 于2019年10月30日周三 下午4:05写道： > Hi Franke, > > From the information provided by Alex: > > >> mvn build jar include com.mysql.jdbc.Driver. > > it seems he has packaged a fat

Flink 1.5 batch job fails to start

Trying to migrate existing job that runs fine on Flink 1.3.1 to Flink 1.5 and getting a weird exception. Job reads json from s3a and writes parquet files to s3a with avro model. Job is uber jar file built with hadoop-aws-2.8.0 in order to have access to S3AFileSystem class. Fails here https://git

Re: Flink 1.5 batch job fails to start

FO Trying to start actor system at flink-jobmanager:6123 2018-07-24T12:09:40.335+ [org.apache.flink.runtime.entrypoint.ClusterEntrypoint] INFO Actor system started at akka.tcp://flink@flink-jobmanager:6123 On Tue, Jul 24, 2018 at 7:16 AM Till Rohrmann wrote: > Hi Alex, > > I'

Re: Flink 1.5 batch job fails to start

ore On Tue, Jul 24, 2018 at 10:32 AM vino yang wrote: > Hi Alex, > > Based on your log information, the potential reason is Hadoop version. To > troubleshoot the exception comes from different Hadoop version. I suggest > you match the both side of Hadoop version. > > You can

Re: Flink 1.5 batch job fails to start

mbda$submitRequest$3(RestClient.java:293) at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ... 5 more Thanks, Alex On Wed, Jul 25, 2018 at 2:22 AM Till Rohrmann wrote: > Hi Alex, > > could you share with us the full logs of the client and the cluster > entrypoint? That would be tre

Re: Flink 1.5 batch job fails to start

problem. And one run of the job actually was successful., but it started failing after that with the same problem. Weird. Will continue to poke around, feels I am so close :) Best, -Alex On Tue, Jul 24, 2018 at 9:08 PM vino yang wrote: > Hi Alex, > > Is it possible that the data has been

Fast serialization for Kotlin data classes

Hi there, I appreciate the fact that Flink has built-in support for making POJO and Scala `case class` serialization faster, but in my project we use immutable Kotlin `data class`es (analogous to Scala `case class`es) extensively, and we'd really prefer not to make them POJOs, mostly for style/tas

Reset of transient variables in state to default values.

seconds and first occurence of this problem is also 5 seconds), otherwise I don't have any clues to a possible explanation. Is it an expected side effect of checkpoint procedure ? Best regards, Alex

Re: Reset of transient variables in state to default values.

dition of the boolean variable isFileOpened, however it's awkward because I need to check the state of the transient variable every time I use state.value(). So should it be expected that transient variables in state would be resetted to default values ? пн, 11 окт. 2021 г. в 12:33, JING ZHANG :

Re: Reset of transient variables in state to default values.

amp; store data for a while, so at least once checkpoint would be triggered. I used a checkpoint interval 5000ms on a quite slow system. Every data chunk is about 1k. Utils.startWriteToPcap - new RandomAccessFile() Utils.writeToPcap - should be replaced with currentFile.write() пн, 11 окт. 2021 г.

Re: Reset of transient variables in state to default values.

ry an alternative approach - I will add my own writeObject implementation, it should work :) Best regards, Alex вт, 12 окт. 2021 г. в 15:07, JING ZHANG : > Hi Alex, > Since you use `FileSystemStateBackend`, I think currentFile became > nullified once in a while is not caused by peri

Re: Reset of transient variables in state to default values.

Hi Arvid, It sounds like a good direction, do I need to register my state class with KryoSerializer , similar to this ? env.getConfig().registerTypeWithKryoSerializer(IPSessionOrganizer.proto.SourceOutput.class, ProtobufSerializer.class); пн, 18 окт. 2021 г. в 10:32, Arvid Heise : > Hi A

How to prevent check pointing of timers ?

Dear flink user, In our project, restoring the timer's state creates numerous issues, so I would like to know if it is possible to avoid save/restore of timers altogether. If it isn't possible, how could I delete all registered timers during the open function ? Best regards, Alexander

Re: How to prevent check pointing of timers ?

tch aggregation [1] solve your problem? > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/tuning/#minibatch-aggregation > > Alex Drobinsky 于2022年2月3日周四 20:41写道： > >> Dear flink user, >> >> In our project, restoring the timer'

Re: How to prevent check pointing of timers ?

er. Second solution is rather hypothetical e.g. somehow forcing the timer to be volatile or reset timer after restore , so the question is if this second solution is feasible ? вт, 8 февр. 2022 г. в 04:19, Yun Tang : > Hi Alex, > > I think the better solution is to know what the problem you h

Re: How to prevent check pointing of timers ?

y to deal with "immediate trigger" phenomenon is to implement workaround at level of onTimer function. Thanks for your time everybody ! Much appreciated, Alex вт, 8 февр. 2022 г. в 09:49, Frank Dekervel : > Hello, > > I guess you already evaluated moving to event time and you w

Re: Cannot configure akka.ask.timeout

Hi there, Run into the same problem running a batch job with Flink 1.6.1/1.6.2 . akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#202546747]] after [1 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". akka

Re: Cannot configure akka.ask.timeout

Hi Qi, Thanks for looking into this. Here is ticket https://issues.apache.org/jira/browse/FLINK-11143 Best, -Alex On Tue, Dec 11, 2018 at 8:47 PM qi luo wrote: > Hi Alex and Lukas, > > This error is controlled by another RPC timeout (which is hard coded and > not affected by “akka

Re: Cannot configure akka.ask.timeout

, -Alex On Wed, Dec 12, 2018 at 7:47 AM Alex Vinnik wrote: > Hi Qi, > > Thanks for looking into this. Here is ticket > https://issues.apache.org/jira/browse/FLINK-11143 > > Best, > -Alex > > On Tue, Dec 11, 2018 at 8:47 PM qi luo wrote: > >> Hi Alex and Luka

Re: Cannot configure akka.ask.timeout

); JsonLinesInputFormat jsonInputFormat = new JsonLinesInputFormat(new Path(inputPath), configuration); jsonInputFormat.setFilesFilter(new BucketingSinkFilter()); DataSet input = env.readFile(jsonInputFormat, inputPath).withParameters(configuration); On Wed, Dec 12, 2018 at 8:53 PM qi luo wrote: > Hi A

AskTimeoutException

creating the local environment. I am passing 2 minutes, bu the error message says it timed out after 10 seconds. Best regards, Alex soto

Re: AskTimeoutException

-199361569]] after [1 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". Best regards, Alex soto > On Apr 12, 2019, at 6:34 PM, Abdul Qadeer wrote: > > Hi Alex, > > The timeout shown in the exception is due to

Re: AskTimeoutException

olution is to upgrade to 1.7.3 or 1.8.0 Best regards, Alex soto > On Apr 24, 2019, at 1:39 PM, Alex Soto wrote: > > Thanks Abdul for the help. So I added this: > > cfg.setString(AkkaOptions.LOOKUP_TIMEOUT, "2 min"); > > > But I am still

unsubscribe

Please unsubscribe me

Reading files from an S3 folder

Hi, I've been playing around with using apache flink to process some data, and I'm starting out using the batch DataSet API. To start, I read in some data from files in an S3 folder: DataSet records = env.readTextFile("s3://my-s3-bucket/some-folder/"); Within the folder, there are 20 gzipped fi

Re: Reading files from an S3 folder

hould not be > read multiple times by the different parallel instances. > How did you check / find out that each node is reading all the data? > > Regards, > Robert > > On Tue, Nov 22, 2016 at 7:42 PM, Alex Reid > wrote: > >> Hi, I've been playing around with using a

akka.remote.OversizedPayloadException

with the command line client: flink run -m … No matter what I do, I always get the same error message with the same default limit. Please help, I am not finding any information on how to override the default limit. Best regards, Alex soto

Re: put record to kinesis and then trying consume using flink connector

record because you told it to start reading from the time you started up the consumer app. "LATEST" does not mean read the last event that was pushed into the stream, it means start reading from "now"/consumer app start basically. - alex On Tue, Apr 25, 2017 at 4:00 PM, Sathi Chow

Re: Flink Checkpoint times out with checkpointed data size doubles every checkpoint.

the state in a big window, e.g. in a 2 hour Tumbling window the maximum job state will be reached in two hours only. So your job should be scaled or optimized. Best Alex On Tue, Jun 20, 2023 at 10:39 AM Prabhu Joseph wrote: > Hi, > > Flink Checkpoint times out with checkpointed data

Re: Request-Response flow for real-time analytics

This is a pretty hard problem. I would be inclined to try Queryable State ( https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/) first. -0xe1a On Mon, Aug 21, 2023 at 11:04 PM Jiten Pathy wrote: > Hi, > We are currently evaluating Flink for

Issues using savepoint upgradeMode with Kubernetes Operator

entually hits my restart limit and then reboots the job. Turning on DEBUG logs doesn't show much either - it just seems it cannot reliably submit savepoints, failing with that generic error. Note that checkpoints work perfectly well, and I can use the last-state upgrade option. - Alex

Kubernetes HA having issues when restarting job

My job in Kubernetes periodically fails with the following error: 2023-10-13 18:22:32,153 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error occurred in the cluster entrypoint. java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException

What to do about local disks with RocksDB with Kubernetes Operator

The recommended practice for RocksDB usage is to have local disks accessible to it. The Kubernetes Operator doesn’t have fields related to creating disks for RocksDB to use. For instance, say I have maxParallelism=10 but parallelism=1. I have a statically created PVC named “flink-rocksdb”. The

Re: Event stuck in the Flink operator

Can you share your precise join semantics? I don't know about Flink SQL offhand, but here are a couple ways to do this when you're using the DataStream API: * use the Session Window join

Re: Global connection open and close

The classic connection pool pattern, where expensive connections are created relatively few times and used by lots of transient short-lived tasks, each of which borrows a connection from the pool and returns it when done, would still be usable here, but as Péter points out, you can't rely on a sing

Fwd: Global connection open and close

[previous didn't cc list, sorry for dupes] The classic connection pool pattern, where expensive connections are created relatively few times and used by lots of transient short-lived tasks, each of which borrows a connection from the pool and returns it when done, would still be usable here, but a

Re: Optimize exact deduplication for tens of billions data per day

It may not be completely relevant to this conversation in this year, but I find myself sharing this article once or twice a year when opining about how hard deduplication at scale can be. 😅 -0xe1a On Thu, Apr 11, 2024 at 10:22 PM Péter Váry wrote

Re: Flink 1.17.2 on K8s infinite TM request

For those who would look for an answer, the fix is available in 1.18: https://issues.apache.org/jira/browse/FLINK-31498 Proposed solution is not to request for TaskManagers if there are some slots already pending. On Thu, Jul 4, 2024 at 2:00 PM Alex Nitavsky wrote: > Hello community, > &g

Flink job can't complete initialisation because of millions of savepoint file reads

We have an issue where a savepoint file containing Kafka topic partitions offsets is requested millions of times from AWS S3. This results in the job crashing and then followed by a restart and crashing again. We have tracked the high number of reads (~3 millions) to Kafka topic partitions (~40k) m

Re: Deployment Rollback Pattern with Flink K8S Operator — Looking for Feedback

Just to add. Even in your custom rollback mechanism in order to rollback you can check the last validated spec field to get the proper image id. On Sun, 27 Apr 2025 at 21:07, Alex Nitavsky wrote: > Apriori your solution make sense. > Just keep in mind that some form of blue green deployme

Re: Deployment Rollback Pattern with Flink K8S Operator — Looking for Feedback

that verification logic could be plugged. On Sun, 27 Apr 2025 at 21:05, Ehud Lev wrote: > Hi Alex, > > Thanks for the response! > > Yes, we did consider the "Application upgrade rollbacks (Experimental)" > feature. > However, we decided not to use it mainly for t

Re: Deployment Rollback Pattern with Flink K8S Operator — Looking for Feedback

feature request. We rely on similar in house mechanism to make more complex verifications. Regards Alex On Sun, 27 Apr 2025 at 20:44, Ehud Lev wrote: > Hi Flink users, > > We have a few Flink topologies running in production, managed by the Flink > Kubernetes Operator, and we typi

#FlinkVectors: how can I attach a label or tag to a vector in ML (batch) for future cross-reference?

Say I’m doing document retrieval on a large batch of docs, and I’m using KNN to predict topics. I noticed however I struggled a bit to look back at the original text documents based on the numerical results only: I had to use some hashing tricks which weren’t optimal. Is it possible to request a

Re: Http Requests from Flink

convert the micro-batches into a stream. I’d love to know about other alternatives. Cheers, Alex From: Ulf Thomas Reply-To: "user@flink.apache.org" Date: Thursday, March 2, 2017 at 12:58 PM To: "user@flink.apache.org" Subject: Http Requests from Flink Hello, I&