Re: TTL in pyflink does not seem to work

which case one possible workaround is to use the RocksDB state > backend instead. Another workaround would be to rely on timers in the process > function, and clear the state yourself. > > David > > > On Fri, Mar 8, 2024 at 12:29 AM lorenzo.affetti.ververica.com via user &

Re: TTL in pyflink does not seem to work

Note, that in Java code, it prints `State: Null`, `State: Null`, as I was expecting in, unlike pyflink code On Mar 7, 2024 at 15:59 +0400, Ivan Petrarka , wrote: > Hi! I’ve created a basic pyflink pipeline with ttl and it does not seem to > work. I have reproduced the exact same code in Ja

TTL in pyflink does not seem to work

Hi! I’ve created a basic pyflink pipeline with ttl and it does not seem to work. I have reproduced the exact same code in Java and it works! Is this a pyflink bug? If so - how can I report it? If not - what can I try to do? Flink: 1.18.0 image: flink:1.18.0-scala_2.12-java11 Code to reproduce.

Re: [EXTERNAL] TaskManagers Crushing

Were you ever able to find a workaround for this? I also have transient failures due to org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException. From: Kenan Kılıçtepe Sent: Saturday, August 19, 2023 5:50 PM To: user@flink.apache.org Subject:

RE: MSI Auth to Azure Storage Account with Flink Apache Operator not working

new to using Flink, and it took me a while to figure out this; but hopefully it is helpful to you. I get the sense that few people are using ADLS with newer Flink versions or something because the docs and support seem half-baked. Let me know if you make progress using MSI. Best of luck, Ivan

RE: MSI Auth to Azure Storage Account with Flink Apache Operator not working

that is somehow helpful. Best of luck, Ivan From: DEROCCO, CHRISTOPHER<mailto:cd9...@att.com> Sent: Monday, May 8, 2023 3:40 PM To: Shammon FY<mailto:zjur...@gmail.com> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: [EXTERNAL] RE: MSI Auth to Azure Storage

Failed docker-entrypoint.sh in Flink-Operator Logs

f they are just something to ignore, it might be good to add an argument to `/docker-entrypoint.sh` to skip running it so there aren’t confusing error messages. Thanks, Ivan

RE: [EXTERNAL] Re: Unable to Use spec.flinkVersion v1_17 with Flink Operator

starting. I will follow-up if I find any new problems. Thanks for the help! Best, Ivan From: Gyula Fóra<mailto:gyula.f...@gmail.com> Sent: Tuesday, March 28, 2023 10:25 PM To: Ivan Webber<mailto:ivan.web...@microsoft.com> Cc: Ivan Webber via user<mailto:user@flink.apache.org>

Unable to Use spec.flinkVersion v1_17 with Flink Operator

es-operator flink-operator-repo/flink-kubernetes-operator helm repo rm flink-operator-repo helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.4.0/ helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator ``` Thanks, Ivan Webber

RE: [EXTERNAL] Re: Secure Azure Credential Configuration

_Identity>, but got an error [1]. If anyone has gotten that to work I’d be interested in hearing about it. Thanks for the help so far; please, anyone who can give pointers send them. Thanks, Ivan [1] - org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azure.AzureException: No credentia

Re: [EXTERNAL] Re: Secure Azure Credential Configuration

Thanks Alexis, I will be trying that out today. If it works I will share back and try adding it to the docs. From: Alexis Sarda-Espinosa Sent: Thursday, March 2, 2023 3:33:03 PM To: Ivan Webber Cc: user Subject: Re: [EXTERNAL] Re: Secure Azure Credential

Secure Azure Credential Configuration

that’d be ideal. Thanks, and sorry for bothering everyone. I’ve just exhausted myself and am hopeful someone will have mercy for me. I really will Venmo you $5 for coffee if you want. Thanks, Ivan Larger code examples: Setting dynamic properties before executing the job doesn’t work becau

Field names must be unique. Found duplicates

I am getting the *Field names must be unique. Found duplicates* error when aggregating over the descriptor columns in HOP windowing function. A full description of the error can be found here . Is there a way to use the descriptor column in aggrega

Just failed while starting

e one large job, which is 10 time more parallelisms then other smaller jobs. Can someone help me on what the root cause of the issue and how to avoid it. Thanks, Ivan

Re: TaskManager crash after cancelling a job

job cancellation routine? Want to understand the logic there. Thanks, Ivan > On Jul 26, 2021, at 7:22 PM, Yangze Guo wrote: > > Hi, Ivan > > My gut feeling is that it is related to FLINK-22535. Could @Yun Gao > take another look? If that is the case, you can upgrade to

TaskManager crash after cancelling a job

. Thanks you, hoping to get some insight knowledge here. Ivan

Re: Flink Kubernetes HA

Thanks for the reply. Yes, We are seeing all the completedCheckpoint and they keep growing. We will revisit our k8s set up, configmap etc > On Jun 23, 2021, at 2:09 AM, Yang Wang wrote: > > Hi Ivan, > > For completedCheckpoint files will keep growing, do you mean to

Flink Kubernetes HA

not keep the all the completeCheckpoint* in HA? Thanks, Ivan

Re: Exception on s3 committer

and restarted. The file is committed on s3 successfully, but not acknowledge recorded on Flink side. In between, the batch job consumed the file. I don’t know if that’s possible. Thanks Ivan > On Aug 30, 2020, at 11:10 PM, Yun Gao wrote: > > > Hi Ivan, > >I think t

Exception on s3 committer

StreamingFileSink/BulkFormat to various s3 prefixes. Thanks in advance. -Ivan 2020-08-28 15:17:58 java.io.IOException: Recovering commit failed for object cbd/late-landing/event_date=2020-08-28/event_hour=16/part-5-264804. Object does not exist and MultiPart Upload

Re: sporadic "Insufficient no of network buffers" issue

Yes, increase the taskmanager.network.memory.fraction in your case. Also reduce the parallelism will reduce number of network buffer required for your job. I never used 1.4.x, so don’t know about it. Ivan > On Jul 31, 2020, at 11:37 PM, Rahul Patwari > wrote: > > Thanks for your

Re: sporadic "Insufficient no of network buffers" issue

your job need the shuffling, then you may consider to add more memory to TM. Thanks, Ivan > On Jul 31, 2020, at 2:02 PM, Rahul Patwari wrote: > > Hi, > > We are observing "Insufficient number of Network Buffers" issue Sporadically > when Flink is upgraded from 1.4.2

Re: Flink 1.11 job stop with save point timeout error

have issue on “flink stop”. Thanks Ivan > On Jul 24, 2020, at 5:15 AM, Robert Metzger wrote: > > Hi Ivan, > thanks a lot for your message. Can you post the JobManager log here as well? > It might contain additional information on the reason for the timeout. > > On Fri,

Flink 1.11 job stop with save point timeout error

ableFuture.java:1928) at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:493) ... 9 more flink@flink-jobmanager-fcf5d84c5-sz4wk:~$ Thanks in advance, Ivan

Completed Job List in Flink UI

I am using Flink 1.9 Thank you in advance. Ivan

Flink on Kubernetes

more nodes/pods (task manager containers) to the cluster, will a running Flink job redistribute load to the additional resources or I have to stop to a savepoint, and restart the job? Thanks and regards. Ivan

Re: Developing Beam applications using Flink checkpoints

Perfect, thank you so much Arvid, I'd expect more people using Beam on top of Flink, but it seems is not so popular. On Tue, 2020-05-19 at 12:46 +0200, Arvid Heise wrote: > Hi Ivan, > > I'm fearing that only a few mailing list users have actually deeper > Beam experience. I

Re: Developing Beam applications using Flink checkpoints

result, so that is doing a double serialization, does it make sense? Or how does it work? On Tue, 2020-05-19 at 08:54 +, Ivan San Jose wrote: > Yep, sorry if I'm bothering you but I think I'm still not getting > this, > how could I tell Beam to tell Flink to use that serializ

Re: Developing Beam applications using Flink checkpoints

we might break other things that Beam is hidding for us... On Tue, 2020-05-19 at 10:44 +0200, Arvid Heise wrote: > Hi Ivan, > > The easiest way is to use some implementation that's already there > [1]. I already mentioned Avro and would strongly recommend giving it > a go. If

Re: Developing Beam applications using Flink checkpoints

0/dev/stream/state/custom_serialization.html#the-typeserializersnapshot-abstraction ) there. Could you please give us some hints? Thanks On Mon, 2020-05-18 at 12:41 +0200, Arvid Heise wrote: > Hi Ivan, > > First let's address the issue with idle partitions. The solution is > to us

Developing Beam applications using Flink checkpoints

Hi, we are starting to use Beam with Flink as runner on our applications, and recently we would like to get advantages that Flink checkpoiting provides, but it seems we are not understanding it clearly. Simplifying, our application does the following: - Read meesages from a couple of Kafka topic

Flink performance tuning on operators

ackend = filesystem. Checkpoint = s3 bucket. Our event object has 300+ attributes. Thanks Ivan

Re: assign time attribute after first window group when using Flink SQL

w2. I'd like to compare the "price" field in every raw event with the same field in the most close preceding event in w2. If condition meets, I'd like to use the price value and timestamp in that event to get one matching event from another raw stream (r2). CEP sounds to be

Re: Is Flink able to do real time stock market analysis?

code hard to read. I’m thinking about watermark, but not sure how to do this. -- Thanks Ivan From: TechnoMage Date: Thursday, 12 April 2018 at 3:21 AM To: Ivan Wang Cc: "user@flink.apache.org" Subject: Re: Is Flink able to do real time stock market analysis? I am new to Flink so

Is Flink able to do real time stock market analysis?

Hi all, I've spent nearly 2 weeks trying to figure a solution to my requirement as below. If anyone can advise, that would be great. 1. There're going to be 2000 transactions per second as StreamRaw, I'm going to tumbleWindow them as StreamA, let's say every 15 seconds. Then I'm going to countWin

assign time attribute after first window group when using Flink SQL

alNode.scala:149) at org.apache.flink.table.plan.logical.WindowAggregate.validate(operators.scala:658) at org.apache.flink.table.api.WindowGroupedTable.select(table.scala:1159) at org.apache.flink.table.api.WindowGroupedTable.select(table.scala:1179) at minno.gundam.ReadPattern.main(ReadPattern.java:156) Is there any way to assign time attribute after the first groupBy (w1)? Thanks Ivan

AvroParquetWriter may cause task managers to get lost

. Bellow there is a code snippet of the class which uses the AvroParquetWriter. Best regards, Ivan Budincevic Software engineer, bol.com Netherlands package com.bol.measure.timeblocks.files; import com.bol.measure.timeblocks.measurement.SlottedMeasurements; import

Re: is there ways to enable checkpoint from flink-conf.yaml?

anymore in an upcoming version. On 26.07.2017 16:15, Ivan wrote: Hi , Flink users we are using Flink as the runtime of our beam jobs which works great, recently we want to enable restart strategy <https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/restart_strategies.html> in

is there ways to enable checkpoint from flink-conf.yaml?

Hi , Flink users we are using Flink as the runtime of our beam jobs which works great, recently we want to enable restart strategy in our flink cluster, from the document I see restart strategy will only

Re: How to get top N elements in a DataSet?

Hi @Fabian, @Gabor, and @Aljoscha, Thank you for your help! It works as expected. Best regards, Ivan. On Tue, 24 Jan 2017 at 17:04 Fabian Hueske wrote: > Aljoscha, you are right. > The second mapPartition() needs to have parallelism(1), but the > sortPartition() as well: >

How to get top N elements in a DataSet?

every partition and find top 10 global values. Best regards, Ivan.

PLC/Scada/Sensor anomaly detection

Hello! Has anyone used Flink in "production" for PLC's sanomaly detections? Any pointers/docs to check? Best regards, Iván Venzor C.