Re: Flink on YARN - tmp directory

I should also note that the above steps did get the Flink JobManager and TaskManagers to save their tmp web dashboard files to /my/tmp/ and to show in the Dashboard that the taskmanager.tmp.dirs property had been properly set to /my/tmp/, but the tmp files I wrote in my jobs stubbornly wrote to /tm

Flink on YARN - tmp directory

Hi, My jobs create tmp files like so: java.nio.file.Path tmpFilePath = java.nio.file.Files.createTempFile("tmpFile", "txt"); They currently appear in /tmp/, but I want them somewhere else, say /my/tmp/ . The Flink on YARN docs say: Flink on YARN will overwrite the following configuration param

Re: Fink: KafkaProducer Data Loss

Hi Gordon, I was able to reproduce the data loss on standalone flink cluster also. I have stripped down version of our code with here: Environment: Flink standalone 1.3.0 Kafka 0.9 *What the code is doing:* -consume messages from kafka topic ('event.filter.topic' property in application.propertie

Re: Operations dependencies between values with different key in a ConnectedStreams

Hi Gabriele, I think CEP may be able to deal with this kind of expressions, in general, although I am not sure about how to deal with different time windows (5s and 3s, in your case). Take a look at the available patterns in the CEP library doc: https://ci.apache.org/projects/flink/flink-docs

Re: CEP condition expression and its event consuming strategy

Hi Dawid, Thank you. Ad. 1 I noticed that the method getEventsForPattern() returns an Iterable and we need to further invoke .operator().next() to get access to the event value. Ad. 2 Here is a bit about a use case we have that calls for such discarding semantics. In the event processing pr

Access Sliding window

Hi, I am using a sliding window to monitor server performance. I need to keep track of number of HTTP requests generated and alert the user when the requests gets too high(Sliding window of 6 hours which slides every 15mins). Aggregate count of the number of http requests is evaluated in the 15mi

Operations dependencies between values with different key in a ConnectedStreams

Hi guys, I have a question for you. I have an application with two keyed data streams: one for control and the other one for the data. Each control message represents an operation to be performed on the data values marked with a certain identifier. I connected the two streams and I process the

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

On Fri, Jul 28, 2017 at 4:03 PM, Stephan Ewen wrote: > Seems like no one raised a concern so far about dropping the savepoint > format compatibility for 1.1 in 1.4. > > Leaving this thread open for some more days, but from the sentiment, it > seems like we should go ahead? +1

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

Seems like no one raised a concern so far about dropping the savepoint format compatibility for 1.1 in 1.4. Leaving this thread open for some more days, but from the sentiment, it seems like we should go ahead? On Wed, Jul 12, 2017 at 4:43 PM, Stephan Ewen wrote: > Hi users! > > Flink currently

Flink QueryableState with Sliding Window on RocksDB

Hi, We recently moved from Spark Streaming to Flink for our stream processing requirements in our organization and we are in the process of removing the number of external calls as much as possible. Earlier we were using HBASE to store the incoming data, but we now want to try out stateful operati

Is watermark used by joining two streams

Hello, i want to join two streams based on event time window, every stream has its own watermark, one has priodic watermark and the other has punctuated watermark. are the watermarks used to trigger the join? if yes, which one and how is it used? Thank you and best regards Wei

Re: re: How can I submit a flink job to YARN/Cluster from java code?

The RemoteEnvironment won’t allow you to spawn a Flink cluster running on Yarn. But you can start one with yarn-session.sh. The same applies to the REST endpoint. You first have to start a cluster. Cheers, Till ​ On Fri, Jul 28, 2017 at 12:14 PM, 程骥 wrote: > Hi, > Thanks for yor help. > This is

??????re: How can I submit a flink job to YARN/Cluster from java code?

Hi, Thanks for yor help. This is a good idea. Maybe I didn't understand the information of restful API. I think the restful API can not submit a job to yarn. Is that right? -- -- ??: "z...@zjdex.com";; : 2017??7??28??(??) 5:15

回复： How can I submit a flink job to YARN/Cluster from java code?

Hi,Till Actually,to submit a flink job to flink clusetr with ExecutionEnvironment.createRemoteEnvironment,I have implemented. but how to submit flink job to flink on yarn,I have no idea. I think this API not supported. Thanks for your help. :) -- 原始邮件 -- 发件人

re: How can I submit a flink job to YARN/Cluster from java code?

Hi: I think you can use Flink restful API, like "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true" The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitorin

Re: How can I submit a flink job to YARN/Cluster from java code?

Hi, it is possible to submit a job to a running cluster (Standalone/Mesos/Yarn) by using the RemoteEnvironment which you can create via ExecutionEnvironment.createRemoteEnvironment and then executing your program (e.g. running it in the IDE). This assumes that the remote cluster is already running

Re: Flink CLI cannot submit job to Flink on Mesos

Hi Francisco, have you set the right high-availability configuration options in your client configuration as described here [1]? If not, then Flink is not able to find the correct JobManager because it retrieves the address as well as a fencing token (called leader session id) from the HA store (Z

Re: Memory Leak - Flink / RocksDB ?

Hi, I see that matching the RocksDB configuration to fit certain container sizes can be very tedious and error prone for users. I have opened a jira to start improving the situation: https://issues.apache.org/jira/browse/FLINK-7289 . Please feel

How can I submit a flink job to YARN/Cluster from java code?

Hello?? I want to submit a flink job to YARN/Cluster from java code.If this is feasible? Is there anyone tried to do it before ?? Thanks