Flink on Kubernetes

Hi, I have setup Filnk 1.9.1 on Kubernetes on AWS EKS with one job manager pod, 10 task manager pods, one pod per EC2 instance. Job runs fine. After a while, for some reason, one pod (task manager) crashed, then the pod restarted. After that, the job got into a bad state. All the parallelisms a

Re: How do I get the IP of the master and slave files programmatically in Flink?

Hi all, I would like to have the IP of the JobManager, not the Task Executors. I explain why. I have an operator (my own operator that extends AbstractUdfStreamOperator) that sends and receives messages from a global controller. So, regardless of which TaskManager these operator instances are dep

Re: Stream Iterative Matching

Hi Guowei, Thank you for your reply. Are you able to give some detail on how that would work with the per window state you linked? I'm struggling to see how the logic would work. I guess something like a session window on a keyed stream (keyed by sensor ID). Timers would fire 90 seconds after eac

Performance impact of many open windows at the same time

Hi all, I'm looking into what happens when messages are ingested with timestamps far into the future (e.g. due to corruption or a wrong clock at the sender). I'm aware of the effect on watermarking, but another thing I'm concerned about is the performance impact of the extra windows this will cre

Re: Using Queryable State within 1 job + docs suggestion

Hi, Thanks for your response! What if I'm using regular state instead of windowState, is there any way to use query this state of a PU from another PU in the same Flink job? Best, Annemarie -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink on Kubernetes

Hi lvan Yang, #1. If a TaskManager crashed exceptionally and there are some running task on it, it could not join back gracefully. Whether the full job will be restarted depends on the failover strategies[1]. #2. Currently, when new TaskManagers join to the Flink cluster, the running Flink job co

Re: Flink Window with multiple trigger condition

Session window defined on the gap of inactivity, I do not have that requirement. Start the window only on the "*search even*t" that part I will take later. Let's say in the first phase I want to start the window on any event that appears for that user. For example : *Scenario -1* t1 - user1

Re: Using Queryable State within 1 job + docs suggestion

Hi Annemarie Actually, I do not know what exactly PU means in your thread. If you means the task manager, though I haven't tried I think we might be able to query state in the same job. Maybe you could give a try. In general, we would initialize two states in the same operator so that they cou

Re: Stream Iterative Matching

Hi, Marc 1. I think you should choose which type of window you want to use first. (Thumbling/Sliding/Session) From your description, I think the session window maybe not suit your case because there is no gap. 2. >>> how this would work in practise or how to handle the case where timers fire for da

Re: Using Queryable State within 1 job + docs suggestion

Hi, So what I meant was that I have a keyed stream, and from each thread/keygroup/PU I want to query the state of the other threads/keygroups/PUs. Does anybody have any experience with this? I'm currently working on it, and the main problem seems to be that the Queryable State Client requires

kerberos integration with flink

Hi guys, Is there any difference in providing kerberos config to the flink jvm using this method in the flink configuration? env.java.opts: -Dconfig.resource=qa.conf -Djava.library.path=/usr/mware/flink-1.7.2/simpleapi/lib/ -Djava.security.auth.login.config=/usr/mware/flink-1.7.2/Jaas/kafka-jaas.

Re: [EXTERNAL] Re: Memory growth from TimeWindows

For those who are interested or googling the mail archives in 8 months, the issue was garbage collection related. The default 1.8 jvm garbage collector (parallel gc) was being lazy in its marking and collection phases and letting the heap build to a level that was causing memory exceptions and

Stateful functions Harness

Hi, I am trying to run https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example locally using https://github.com/apache/flink-statefun/tree/master/statefun-

Re: Stateful functions Harness

Also, where do I put flint-conf.yaml in Idea to add additional required config parameter: classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf > On May 21, 2020, at 12:22 PM, Boris Lublinsky > wrote: > > Hi, > I am trying to run > htt

[no subject]

Re: How do I get the IP of the master and slave files programmatically in Flink?

Hi, Felipe I see your problem. IIUC, if you use AbstractUdfStreamOperator, you could indeed get all the configurations(including what you defined in flink-conf.yaml) through "AbstractUdfStreamOperator#getRuntimeContext().getTaskManagerRuntimeInfo().getConfiguration()". However, I guess it is not t

Re: kerberos integration with flink

Hi, Nick, >From my understanding, if you configure the "security.kerberos.login.keytab", Flink will add the AppConfigurationEntry of this keytab to all the apps defined in "security.kerberos.login.contexts". If you define "java.security.auth.login.config" at the same time, Flink will also keep the

Re: Stateful functions Harness

Hi, On Fri, May 22, 2020 at 7:19 AM Boris Lublinsky < boris.lublin...@lightbend.com> wrote: > Also, where do I put flint-conf.yaml in Idea to add additional required > config parameter: > > classloader.parent-first-patterns.additional: > org.apache.flink.statefun;org.apache.kafka;com.google.prot

Re: Stateful functions Harness

Sorry, forgot to cc user@ as well in the last reply. On Fri, May 22, 2020 at 12:01 PM Tzu-Li (Gordon) Tai wrote: > As an extra note, the utilities you will find in `statefun-e2e-tests`, > such as the `StatefulFunctionsAppsContainers` is not yet intended for users. > This however was previously d

Re: Stateful functions Harness

Are you getting an exception from running the Harness? The Harness should already have the required configurations, such as the parent first classloading config. Otherwise, if you would like to add your own configuration, use the `withConfiguration` method on the `Harness` class. On Fri, May 22,

Re: Performance impact of many open windows at the same time

Hi Joe, The main effect this should have is more state to be kept until the windows can be fired (and state purged). This would of course increase the time it takes to checkpoint the operator. I'm not sure if there will be significant runtime per-record impact caused by how windows are bookkeeped

Re: Using Queryable State within 1 job + docs suggestion

Hi, That in general is not a good idea, with the problem you mentioned as well as the fact that the state you query within the same job using queryable state does not provide any means of consistency guarantee. When it comes to "querying state from another operator", it is a hint that your use ca

Re: Using Queryable State within 1 job + docs suggestion

This in general is not a good idea, as the state you query using queryable state within a job does not provide any consistency guarantees at all. Would it be possible to have some trigger that emits state of the windows, and join the states downstream? In general, that is a better approach for wha

Re: Flink Window with multiple trigger condition

Hi, To achieve what you have in mind, I think what you have to do is to use a processing time window of 30 mins, and have a custom trigger that matches the "start" event in the `onElement` method and return TriggerResult.FIRE_AND_PURGE. That way, the window fires either when the processing time h

Re: Stateful functions Harness

Hi, Sorry, I need to correct my comment on using the Kafka ingress / egress with the Harness. That is actually doable, by adding an extra dependency to `statefun-flink-distribution` in your Harness program. That pulls in all the other required dependencies required by the Kafka ingress / egress,

回复：Performance issue when writing to HDFS

Hi Kong, Sorry that I'm not expert of Hadoop, but from the logs and Google, It seems more likely to be a problem of HDFS side [1] ? Like long-time GC in DataNode. Also I have found a similar issue from the history mails [2], and the conclusion should be similar. Best, Yun [1]

Writing to SQL server

Hi, I am trying to write input from Kafka to a SQL server on AWS, but I have difficulties. I get the following error could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[org.apache.flink.types.Row] [error] val dsRow = tableEnv.toAppen

Re: Writing to SQL server

Hi Martin, usually, this error occurs when people forget to add `org.apache.flink.api.scala._` to their imports. It is triggered by the Scala macro that the DataStream API uses for extracting types. Can you try to call `result.toAppendStream[Row]` directly? This should work if you import `or

Why Flink Connector JDBC does't support LocalDateTime ?

Hello, my friends env: Flink 1.10, Blink Planner table source CREATE TABLE josn_table ( order_id VARCHAR, event_time TIMESTAMP(3), proc_time AS PROCTIME() ) WITH ( 'connector.properties.0.key' = 'bootstrap.servers', 'connector.properties.0.value' = 'localhost:9092', 'connector.proper

Re: [ANNOUNECE] release-1.11 branch cut

Hi, Quick update on the release. After the feature freeze me and Zhijiang were reached out by quite a bit of contributors, which had features in "almost done" state and were asking for permission to merge them after the announced release branch cut and in most cases we granted this permission. As

Flink TTL for MapStates and Sideoutputs implementations

public class CartPGCoprocessFunction extends KeyedCoProcessFunction { private static final Logger logger = LoggerFactory.getLogger(CartPGCoprocessFunction.class); /** * Map state for cart messages, orderId+mid is key and cartMessage is value. */ private static MapState car