Re: Flink 1.2.0 - Flink Job restart config not using Flink Cluster restart config.

Hi Vera, Apparently, if there no job-specific restart strategy, an infinite FixedDelayRestartStrategy is always used for the job submission: https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L571-L57

Re: New message processing time after recovery.

Hi! That is correct. With processing time, all time-based operations will use the current machine system time (which would take into account). Note that with processing time, the elements don’t carry a meaningful timestamp. Best, Gordon On 28 June 2017 at 11:22:43 AM, yunfan123 (yunfanfight...

Re: Partition index from partitionCustom vs getIndexOfThisSubtask downstream

Hi Urs, Yes, the returned “index” from the custom partitioner refers to the parallel subtask index. I agree that the mismatching terminology used could be slightly misleading. Could you open a JIRA to improve the Javadoc for that? Thanks! Cheers, Gordon On 27 June 2017 at 10:40:47 PM, Urs Sch

Re: Checkpointing with RocksDB as statebackend

Hi Vinay, We observed a similar problem before. We found that RocksDB keeps a lot of index and filter blocks in memory. With the growth in state size (in our cases, most states are only cleared in windowed streams), these blocks will occupy much more memory. We now let RocksDB put these blocks in

Re: MapR libraries shading issue

Cool. For future reference, I created a JIRA ticket: https://issues.apache.org/jira/browse/FLINK-7033 Thanks for all the help, guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/MapR-l

Re: Checkpointing with RocksDB as statebackend

Hi Aljoscha, I am using event Time based tumbling window wherein the allowedLateness is kept to Long.MAX_VALUE and I have custom trigger which is similar to 1.0.3 where Flink was not discarding late elements (we have discussed this scenario before). The watermark is working correctly because I ha

Re: Checkpointing with RocksDB as statebackend

Hi, Just a quick question, because I’m not sure whether this came up in the discussion so far: what kind of windows are you using? Processing time/event time? Sliding Windows/Tumbling Windows? Allowed lateness? How is the watermark behaving? Also, the latest memory usage graph you sent, is tha

Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator

I had same concern regarding HBase . So I also added metric to measure Hbase op time in flatmap (Basically complete flatmap op). >From metrics I see that aprox 96 % time op time was under 1 sec. (Still I can do a dummy run without HBase op . But did these timing make sense?) -- View this messa

Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator

I see, what I consider highly likely here is that the lookup to HBase is the bottleneck. If the lookup takes to long events “sit in a queue” between the map and flatMap operations. If you replace the HBase lookup by some dummy code you should see the latency go away. The reason you don’t see la

Flink 1.2.0 - Flink Job restart config not using Flink Cluster restart config.

Hi all, We are running Flink 1.2.0. Our flink-conf.yaml is configured to use a default restart-strategy of fixed-delay, with 3 attempts: restart-strategy: fixed-delay restart-strategy.fixed-delay.attempts: 3 These settings are echoed by the GlobalConfiguration (see first set of log statements).

Re: Problem with Summerization

The stacktrace indicates this may be a bug that was fixed for Flink 1.2: [FLINK-4624] [gelly] Support null values in Graph Summarization * Bug was caused by serializers that cannot handle null values (e.g. Long) * VertexGroupItem now uses Either instead of VV * Generalized test ca

Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator

Source is KafKa . FlatMap has HBase Lookup Sink is Kafka . I tried to get stats over the days . I see that almost 40 % were having latency of 0 seconds , 10 % 0-30 sec, approx 10% 30-60 sec and 10 % around 60 - 120 sec and 30 % around 120 - 210 secs . -- View this message in context: http://ap

Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator

I think then there is something going wrong somewhere. Usually people get millisecond latencies even when they have a “keyBy” or shuffle in-between operations (which are not different to a custom partitioner at the system level). What kind of sources/sinks is your program using? Best, Aljoscha

Re: MapR libraries shading issue

I would say that this is a MapR issue. It's a good idea to add it to the docs in case someone else stumbles upon this. Would be great if you could open a JIRA for that. On 27.06.2017 19:35, ani.desh1512 wrote: Again as I mentioned in the MapR thread, So, after some more digging, I found out