test

ignore

Re: Checkpointing with RocksDB as statebackend

Hi Stephan, Just an update, last week I did a run with state size close to 18GB, I did not observe the pipeline getting stopped in between with G1GC enabled. I had observed checkpoint failures when the state size was close to 38GB (but in this case G1GC was not enabled) Is it possible to get the

Java library for Flink-Kudu integration

Hi all, I apologize for sending the email to both user and dev accounts, but not sure where this topic fits better. In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library al

How to rewind Kafka cursors into a Flink job ?

Hi, Is there a way to rewind Kafka cursors (while using here Kafka as a consumer) from (inside) a Flink job ? Use case [plan A] * The Flink job would listen 1 main "data" topic + 1 secondary "event" topic * In case of a given event, the Flink job would rewind all Kafka cursors of the "data" topic

Re: Java library for Flink-Kudu integration

Hi Ruben, thanks for sharing this! A Flink Kudu connector is a great contribution and Bahir seems to be the right place for it. Thanks, Fabian 2017-03-27 15:35 GMT+02:00 : > Hi all, > > I apologize for sending the email to both accounts, but not sure where > this topic fits better. > > In my t

Re: accessing flink HA cluster with scala shell/zeppelin notebook

Hi Robert, Hi Till, I tried to setup high-availibility options in zepplin, but i guess it's just a matter of flink version compatibility on zepplin side. I'll try to compile zepplin with 1.2 and add needed parameter to see if its better. Thanks for your help ! 2017-03-27 15:09 GMT+02:00 Till Rohr

Apache Flink Hackathon

Hi all, My name is Lior and I am working at Parallel Machines (a startup company located in the Silicon Valley). We are hosting a Flink Hackathon on April 10, 3pm - 8pm at Hotel Majestic in San Francisco. (During the first day of Flink Forward, training day) More details at the meet up event: ht

Flink 1.2 time window operation

Hi all, Lately I’ve been investigating onto the performance characteristics of Flink part of our internal benchmark. Part of this we’ve developed and deployed an application that pools data from Kafka, groups the data by a key during a fixed time window of a minute. In total, the topic that t

Figuring out when a job has successfully restored state

Hi all, I am trying to figure out the best way to tell when a job has successfully restored all state and started process. My first idea was to check the rest api and the number of processed bytes for each parallel operator and if thats greater than 0, it started. Unfortunately this logic fails i

回复：question about record

Hi  lining ,  The records would be serialized and wrote into the buffer. And the buffer will be sent out until it is full or exceeds the timeout.So different records may share the same buffer if the serialization result is less than the buffer size, otherwise one record may span multi buffer

Re: Windows emit results at the end of the stream

Do you mean that window results with timestamps larger than the 6th hour won't be output? Can it be the case that some of the keys simply don't have elements arriving after the 6th hour? It's really hard to tell given no knowledge about your input data set. Maybe you can compile a small example cod

Java library for Flink-Kudu integration

Hi all, I apologize for sending the email to both accounts, but not sure where this topic fits better. In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library allows reading/

Re: ClassNotFoundException upon Savepoint Disposal

What kind of state backend where you using for the checkpoints? If there is a bug that prevents us from deleting the savepoint files automatically, we can do a manual workaround and delete the checkpoints files manually. With Flink 1.3 this becomes very straight forward as savepoint data all go to

Re: accessing flink HA cluster with scala shell/zeppelin notebook

Hi Maciek and Alexis, as far as I can tell, I think it is currently not possible to use Zeppelin with a Flink cluster running in HA mode. In order to make it work, it would be necessary to specify either a Flink configuration for the Flink interpreter (this is probably the most general solution) o

Re: Re: flink Broadcast

Hi Rimin, I've just tested a Flink application consisting of multiple jobs similar to yours (using collect) with the `yarn-cluster` option and Flink 1.2.0 and it seemed to work. The yarn cluster was only shut down after the last Flink job has been executed. Could you maybe test if your problem sti

Re: ClassNotFoundException upon Savepoint Disposal

Hi, could you provide us with the log from the job client, with logging on debug level for package org.apache.flink.client? Also, did you check if this problem also exists in the latest bugfix release for your version (1.1.5) ? Best, Stefan > Am 27.03.2017 um 11:41 schrieb Konstantin Gregor

Re: Windows emit results at the end of the stream

The degree of parallelism in the experiments I mentioned is 8. If I decrease the parallelism it emits more results. If I set the parallelism to 1 then it emits results from the entire dataset (i.e., it behaves as expected). What could be the reason of this? -- View this message in context: http

ClassNotFoundException upon Savepoint Disposal

Hey everyone, we are experiencing an issue in the disposal of savepoints in Flink-1.1.3. We have a streaming job that has custom state (user objects are part of the state). We create a savepoint: $ flink savepoint [...] Savepoint completed. Path: hdfs:/bigdata/flink/savepoints/savepoint-20f064fb

question about record

Hi All , data transmission is achieved through the buffer. If record serialised to multiple buffer, whether different records will share the same buffer. Can anyone tell me the answer, Thank!

Re: Flink CEP

Hi Daniel, The NOT operation is not yet supported in the CEP library but there is an open issue https://issues.apache.org/jira/browse/FLINK-3320 and we are working on integrating it in the next release of the CEP library. Please monitor the is