Re: External checkpoint metadata in Flink 1.5.x

Out of curiosity, how will you work around it? And how is it easier for your tooling if checkpoints are in a central location? Best, Aljoscha > On 12. Jul 2018, at 17:55, Gyula Fóra wrote: > > Hi! > > Well it depends on how we look at it FLINK-5627 >

答复: 答复: TumblingProcessingTimeWindow emits extra results for a same window

Hi Hequn, I am using Flink 1.4. The job was running with 1 parallelism. I don’t think the extra records are caused by different keys, because: 1. I ran 2 jobs consuming the same source, jobA with 2-minute window, and job with 4-minute window. If there are wired keys, then jobA will get no m

Re: high availability with automated disaster recovery using zookeeper

I've used a multi-datacenter Consul cluster used to coordinate service-discovery. When a service starts up in the primary DC, it registers itself in Consul with a key that has a TTL that must be periodically renewed. If the service shuts down or terminates abruptly, the key expires and is removed f

Taskmanager SSL fails looking for Subject Alternative IP Address

I have the following SSL configuration for a 3 node HA flink cluster: #taskmanager.data.ssl.enabled: false security.ssl.enabled: true security.ssl.keystore: /opt/app/certificates/server-keystore.jks security.ssl.keystore-password: security.ssl.key-password: security.ssl.truststore: /opt/app/cert

RE: Regarding the use of RichAsyncFunction as a FlatMap

Regarding the ‘partiality’ you mention in option one, wouldn’t you have to give that up anyway to maintain exactly once processing? Suppose input message A results in asynchronous queries/futures B and C, and imagine the following series of events: 1. Your function receives A 2. As

Loading Rules from compacted Kafka Topic - open() vs Connected Streams

Hello All, I can think of two options of implementing below requirement and request some guidance on choosing the option with pros and cons. Requirements: - A in memory rules cache to be loaded from log compacted kafka topic. This cache has to be loaded prior to arrival of events. - Updates to th

Re: Need assistance : creating remote environment

Yep, that is due to version mismatch. Working fine now. Thank you Chesnay Schepler. On Wed, Jul 11, 2018 at 6:05 PM Chesnay Schepler wrote: > Based on the logs your client is using the RestClusterClient, which means > that the client is either > a) running 1.4 with the flip6 profile enabled > b

Re: External checkpoint metadata in Flink 1.5.x

Hi! Well it depends on how we look at it FLINK-5627 is not necessarily the current behaviour. You still can't really specify the exact location from within the job as it now goes to a checkpoint specific place determined by the checkpoint dir. I

Regarding the use of RichAsyncFunction as a FlatMap

I was wondering if there is a way to create an asynchronous flatmap function in Flink. As far as I am aware, the asynchronous function only accepts a single result future as its return (which can be the aggregate list of the flatmap, but then partiality is lost as we have to wait for all of the re

Re: 答复: TumblingProcessingTimeWindow emits extra results for a same window

Hi Yuan, Haven't heard about this before. Which flink version do you use? The cause may be: 1. userId not 100% identical, for example contains invisible characters. 2. The machine clock vibrated. Otherwise, there are some bugs we don't know. Best, Hequn On Thu, Jul 12, 2018 at 8:00 PM, Yuan,Yo

Re: How to create User Notifications/Reminder ?

Thanks a lot Dawid and Hequn. Dawid, the link you provided is very useful. Thanks shyla On Thu, Jul 12, 2018 at 5:59 AM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > Hi shyla, > > It should be doable with CEP. You can create pattern like: > Pattern.begin("start").next/followedBy("end"

Re: How to create User Notifications/Reminder ?

Thanks Hequn and Dawid for your input. Thanks Dawid for the link. Great help! On Thu, Jul 12, 2018 at 5:59 AM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > Hi shyla, > > It should be doable with CEP. You can create pattern like: > Pattern.begin("start").next/followedBy("end").where(.

Re: External checkpoint metadata in Flink 1.5.x

Hi, does that mean https://issues.apache.org/jira/browse/FLINK-5627 is no longer relevant for you, since it seems to request the behaviour that we have now? But yes, I think it's currently not possible (with out-of-box functionality) to write

Re: Flink job hangs using rocksDb as backend

Hi, Did you check the metrics for the garbage collector? Stuck with high CPU consumption and lots of timers sound like there could be a possible problem, because timer are currently on-heap objects, but we are working on RocksDB-based timers right now. Best, Stefan > Am 12.07.2018 um 14:54 sc

Re: How to create User Notifications/Reminder ?

Hi shyla, It should be doable with CEP. You can create pattern like: Pattern.begin("start").next/followedBy("end").where(...).within(/* two weeks*/) and subscribe for timed out events. You can check very similar example here[1]. Best, Dawid [1] https://github.com/dataArtisans/flink-training-exe

Re: Flink job hangs using rocksDb as backend

Thanks Stefan/Stephan/Nico, Indeed there are 2 problem. For the 2nd problem ,I am almost certain that explanation given by Stephan is the true as in my case as there number of timers are in millions. (Each for different key so I guess coalescing is not an option for me). If I simplify my problem,

Re: How to create User Notifications/Reminder ?

Hi shyla, Considering window, I think it is not very convenient. Two weeks window is used to process data in the recent 2 weeks while you want to process data beyond 2 weeks. I'm not familiar with CEP, but it sounds like a good idea. Best, Hequn On Thu, Jul 12, 2018 at 10:56 AM, shyla deshpande

External checkpoint metadata in Flink 1.5.x

Hi, It seems that the behaviour to store the checkpoint metadata files for externalized checkpoints changed from 1.4 to 1.5 and the docs seem to be incorrectly saying that: "state.checkpoints.dir: The target directory for meta data of externalized checkpoints

答复: TumblingProcessingTimeWindow emits extra results for a same window

Hi Timo, This problem happens 4-5 times a day on our online server, with ~15k events per second load, and it is using PROCESSING time. So I don’t think I can stably reproduce the issue on my local machine. The user ids are actually the same, I have doubled checked that. Now, I am wondering coul

Re: flink 1.5 Rest API issues

Thank you for the response. The issue is caused by rest API use comma , as separator which breaks the code. Also, single quote is somehow removed by rest call process. Sent from my iPhone will > On Jul 12, 2018, at 01:05, Gary Yao wrote: > > Hi Will, > > Is the HTTP request hanging, or is th

Re: high availability with automated disaster recovery using zookeeper

Hi Tovi, you can define hard host attribute constraints for the TaskManagers. See the configuration section [1] for more information. If you want to run the JobManager/cluster entry point on Mesos as well, then I recommend starting it with Marathon [2]. This will also give you HA for the master p

Re: State backend configuration

Hi James, the configuration options were consolidates a bit for Flink 1.5. What was called state.backend.fs.checkpointdir before was the same as state.checkpoints.dir, we now made this the same for the two backends. Also, the file-backend does not need local storage on the worker nodes, that's

Re: TumblingProcessingTimeWindow emits extra results for a same window

Hi Yuan, this sounds indeed weird. The SQL API uses regular DataStream API windows underneath so this problem should have come up earlier if this is problem in the implementation. Does this behavior reproducible on your local machine? One thing that comes to my mind is that the "userId"s mig

Re: REST API time out ( flink 1.5 ) on SP

Hi, akka.jvm-exit-on-fatal-error says whether Akka will terminate the JVM if it encounters a fatal error (e.g. OOM). This has nothing todo with quarantining. The option is relevant for the legacy and new (FLIP-6) code base. taskmanager.exit-on-fatal-akka-error is the configuration option which en

State backend configuration

Comparing the state backend configurations between Flink 1.4 and 1.5, I have a question: In Flink 1.5, we have a *state.checkpoints.dir* location and a *state.backend.rocksdb.localdir*. It was the same in Flink 1.4. However, in the case of filesystem backends, we have *state.backend.fs.checkpointd

Re: Flink job hangs using rocksDb as backend

Hi, adding to what has already been said, I think that here can be two orthogonal problems here: i) why is your job slowing down/getting stuck? and ii) why is cancellation blocked? As for ii) I think Stephan already gave to right reason that shutdown could take longer and that is what gets the

TumblingProcessingTimeWindow emits extra results for a same window

Hi community, I have a job which counts event number every 2 minutes, with TumblingWindow in ProcessingTime. However, it occasionally produces extra DUPLICATED records. For instance, for timestamp 153136848 below, it emits a normal result (cnt=1641161), and then followed by a few more recor