Re: [DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

Sounds reasonable to me. If it is a broken feature, then there is not much value in it. On Tue, Apr 23, 2019 at 7:50 PM Gary Yao wrote: > Hi all, > > As the subject states, I am proposing to temporarily remove support for > changing the parallelism of a job via the following syntax [1]: > >

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

Hi Till, IMHO, allow adding hooks involves 2 steps. 1. Provide hook interface, and call these hook in flink (ClusterClient) at the right place. This should be done by framework (flink) 2. Implement new hook implementation and add/register them into framework(flink) What I am doing is step 1 which

Re: Getting JobExecutionException: Could not set up JobManager when trying to upload new version

Might be useful for someone, Regarding this issue. it seems that changing the uid of the operator made this mess . On Tue, Apr 16, 2019 at 6:31 PM Avi Levi wrote: > I am trying to upload a new version of the code but I am getting the > exception below. The schema of the state was not changed for

Re: lack of function and low usability of provided function

Found another function which does not implement the function as it declared. https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/functions.html#temporal-functions The func

Re: Error restoring from checkpoint on Flink 1.8

Hi Congxian, I think I have figured out the issue. It's related to the checkpoint directory collision issue you responded to in the other thread. We reproduced this bug on 1.6.1 after unchaining the operators. There are two stateful operators in the chain, one is a CoBroadcastWithKeyedOperator, t

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

Anyione ? I think there some race condition . These are the TM logs.. I am puzzled b'coz in a larger pipe ( there are about 32 lots on 8 replicas and it works 2019-04-24 01:16:20,889 DEBUG org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager - Releasing local state under allo

Re: Apache Flink - Question about dynamically changing window end time at run time

Hi Mans, I am not sure what you meant by "dynamically change the end-time of a window. If you are referring to dynamically determines the firing time of the window, then it fits into the description of session window [1]: If you want to handle window end time dynamically, one way of which I can th

Re: How to implement custom stream operator over a window? And after the Count-Min Sketch?

Hi Felipe, In a short glance, the question can depend on how your window is (is there any overlap like sliding window) and how many data you would like to process. In general, you can always buffer all the data into a ListState and apply your window function by iterating through all those buffere

get custom gauge metric from WebMonitorEndpoint

I've got custom metric -> [cid:0dbd2582-19b6-4a0e-9dff-49471e973c4c] And I'm using them as suggested in the documentation -> getRuntimeContext().getMetricGroup().gauge("MyCustomMetric", new TestMetric()); I want to get this metric with GET method, but so far I tried almost everything in the

Looking for help in configuring Swift as State Backend

Hi, I am looking for some help in configuring the Swift Fs as State Backend. I am unable to configure it, let me know if anyone has prior done this or knowledge to help me Do we still need to run an HDFS to use this feature ? Thanks, Shakir

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

I see this in the TM pod 2019-04-23 19:08:41,828 DEBUG org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15cc7f3d88466a5 after 0ms 2019-04-23 19:08:47,543 DEBUG org.apache.flink.runtime.taskexecutor.TaskExecutor- Received heartbeat

Re: No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

Though looking at https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/StandaloneResourceManager.java#L88 it does seem that the last log . is expected. Not sure what part is hanging... I have more logs I can share... On Tue, Apr 23, 2019

No zero ( 2 ) exit code on k8s StandaloneJobClusterEntryPoint when save point with cancel...

I am seeing this weird issue where I do a save point with cancel on a job on k8s and it hangs for 5 minutes ( no INFO logs ) and then exits with code of 2. 2019-04-23 17:36:31,372 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Shutting down rest endpoint. 2019-04-23 17:36:

[DISCUSS] Temporarily remove support for job rescaling via CLI action "modify"

Hi all, As the subject states, I am proposing to temporarily remove support for changing the parallelism of a job via the following syntax [1]: ./bin/flink modify [job-id] -p [new-parallelism] This is an experimental feature that we introduced with the first rollout of FLIP-6 (Flink 1.5). Ho

Re: Sinking messages in RabbitMQ

I think you should exchangeDeclare when you open the sink. When invoked, you can channel.basicPublish(exchangeName). Would this work? We have a single exchange, so didn't explore this method. --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www

Re: Fast restart of a job with a large state

Hi Stefan, Paul, Thanks for the tips! Currently I have not tried neither rescaling from checkpoints nor task local recovery. Now it's a subject to test. In case it will be necessary not to just rescale a job, but also to change its DAG - is there a way to have something like let's call it "local

Sinking messages in RabbitMQ

I'm using Flink RabbitMQ Connector for Sinking Data but using the RMQConnectionConfig object I couldn't find any method to set the type of the exchange (Fanout, Topic, Direct). And also the RMQSink get just name of the queue as the parameter. Is there any way to specify the exchange type?

May be useful: our reference document for "Understanding State in Flink"

We keep a document with state-related use cases in our application, useful for onboarding new engineers in the application. See attached PDF. May be useful for others. And of course, corrections are welcome. (Couldn't share our Wiki page) --- Oytun Tez *M O T A W O R D* The World's Fastest Huma

Re: Missing state in RocksDB checkpoints

On Tue, 23 Apr 2019 10:53:52 -0400, Congxian Qiu wrote: > Sorry for the misleading, in the previous email, I just want to say the > problem is not caused by the UUID generation, it is caused by the different > operators share the same directory(because currentlyFlink uses JobVertx as > the direc

Apache Flink - Question about dynamically changing window end time at run time

Hi: I am working on a project and need to change the end time of the window dynamically.  I want to find out if the end time of the window is used internally (for sorting windows/etc) except for handling watermarks that would cause problems if the end time was changed during run time after the w

How to implement custom stream operator over a window? And after the Count-Min Sketch?

Hi, I want to implement my own operator that computes the Count-Min Sketch over a window in Flink. Then, I found this Jira issue [1] which is exactly what I want. I believe that I have to work out my skills to arrive at a mature solution. So, the

Re: Missing state in RocksDB checkpoints

Hi Ning, Sorry for the misleading, in the previous email, I just want to say the problem is not caused by the UUID generation, it is caused by the different operators share the same directory(because currentlyFlink uses JobVertx as the directory) Best, Congxian On Apr 23, 2019, 19:41 +0800, Ning

Re: Missing state in RocksDB checkpoints

Congxian, We just did a test. Separating the two stateful operators from chaining seems to have worked around the problem. The states for both of them are successfully saved in the checkpoint. Ning On Tue, Apr 23, 2019 at 7:41 AM Ning Shi wrote: > > Congxian, > > Thank you for creating the tick

Re: PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

Thank you Guowei and Dawid! I am trying your suggestions today and will report back. - I assume the cleaning operation should be done only once because of the upgrade, or should I run every time the application is up? - `static` sounds a very simple fix to get rid of this. Any drawbacks here?

Re: How to pass application name when using FlinkKinesisConsumer

Hey Liu, Thank you for your response, by saying applicationName, i meant the name to control table in Amazon DynamoDB while creating a consumer. Usually in the non-flink env, we are able to control it by passing the applicationName in the configuration properties when creating a Worker using kines

Re: status on FLINK-7129

+1 On Tue, Apr 23, 2019, 05:18 Vishal Santoshi wrote: > +1 > > On Tue, Apr 23, 2019, 4:57 AM kant kodali wrote: > >> Thanks all for the reply. I believe this is one of the most important >> feature that differentiates flink from other stream processing engines as >> others don't even have CEP y

Apache Flink survey by Ververica

Hi everyone! Ververica is running a brief survey to understand Apache Flink usage and the needs of the community. We are hoping that this survey will help identify common usage patterns, as well as pinpoint what are the most needed features for Flink. We'll share a report with a summary of findin

Flink Customized read text file

Hi, I want to know is it possible to use PipedInutStream and PipedOutputStream in Flink for reading text data from a file? For example extending a RichSourceFunction for it and readata like this: DataStream raw = env.addSource(new PipedSource(file_path)); Actually i tried to implement a class fo

Re: status on FLINK-7129

+1 On Tue, Apr 23, 2019, 4:57 AM kant kodali wrote: > Thanks all for the reply. I believe this is one of the most important > feature that differentiates flink from other stream processing engines as > others don't even have CEP yet. so it would be great if this issue can get > more attention as

Re: Missing state in RocksDB checkpoints

Congxian, Thank you for creating the ticket and providing the relevant code. I’m curious why you don’t think the directory collision is not a problem. What we observe is that one of the operator states are not included in the checkpoint and data is lost on restore. That’s a pretty serious probl

Re: status on FLINK-7129

Thanks all for the reply. I believe this is one of the most important feature that differentiates flink from other stream processing engines as others don't even have CEP yet. so it would be great if this issue can get more attention as I don't think anyone want's to restarts the Job every time the

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

I think we should not expose the ClusterClient configuration via the ExecutionEnvironment (env.getClusterClient().addJobListener) because this is effectively the same as exposing the JobListener interface directly on the ExecutionEnvironment. Instead I think it could be possible to provide a Cluste

HA lock nodes, Checkpoints, and JobGraphs not being removed after failure

originally posted to the dev group, but it's a bit easy for things to get buried a bit there, and this may concern other HA users. Flink v1.7.1 After a Flink reboot we've been seeing some unexpected issues with excess retained checkpoints not being able to be removed from ZooKeeper after a new