Optimal Configuration for Cluster

Hi All, We are trying to running our job in cluster that has this information 1. # of machine: 16 2. memory : 128 gb 3. # of core : 48 However when we try to run we have an exception. "insufficient number of network buffers. 48 required but only 10 available. the total number of network buffers

RE:Flink HA

Thanks for the quick reply ! > state.backend.fs.checkpointdir Is actually pointing to a hdfs directory but I will modify the recovery.zookeeper.path.root > This is only relevant if you are using YARN. From your complete Yes, I omit to say we will use YARN. >Does this help? Yes, a lot :-) Thom

Re: Finding the average temperature

Hi, as I understand it the “temp_reading_timestamp” field is not a key on which you can partition your data. This is a field that would be used for assigning the elements to timestamps. In you data you also have the “probeID” field. This is a field that could be used to parallelize computation,

Re: events eviction

Hi, yes, in some cases it could be necessary. Could you maybe give some example of what kind of window computation you want to achieve? Then we can see if it would be possible without GlobalWindows and evictor. Cheers, Aljoscha > On 15 Feb 2016, at 18:07, Radu Tudoran wrote: > > Hi, > > Thank

RE:Flink HA

After set this configuration, I have some exceptions : java.lang.Exception: Could not restore checkpointed state to operators and functions at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateLazy(StreamTask.java:414) at org.apache.flink.streaming.runtime.tasks.St

Re: Problem with Kafka 0.9 Client

Had the same problem as Javier's. 3450 [Thread-10] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Group metadata response ClientResponse(receivedTimeMs=1455811593680, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consume

Re: Problem with Kafka 0.9 Client

Thank you. Can you send me also the list of properties you are passing to the kafka consumer? Are you only setting the "bootstrap.servers" or more? On Fri, Feb 19, 2016 at 11:46 AM, Lopez, Javier wrote: > Hi Robert, > > Please find attached the full logs of one of our latest executions. We are >

Re: Problem with Kafka 0.9 Client

Hi, these are the properties: Properties properties = new Properties(); properties.setProperty("bootstrap.servers", ".87:9092,.41:9092,.35:9092"); //full IPs removed for security reasons properties.setProperty("zookeeper.connect", ".37:2181"); properties.setProperty("group.

Re: Kyro Intermittent Exception for Large Data

Thanks for the pointer Ken. As far as I know, we’re using the StdInstantiatorStrategy as the fallback instantiator strategy for our Kryo instances. Cheers, Till ​ On Fri, Feb 19, 2016 at 12:39 AM, Ken Krugler wrote: > I've seen this type of error when using Kryo with a Cascading scheme >

Trying to comprehend rolling windows + event time

Hi, I have a setup where I'm feeding a rolling window with event time: https://gist.github.com/lofifnc/dd946fef6f4b3eb25ef1 (Obviously i'm using Flinkspector) The first case behaves as expected I'm emitting three records which are all in the time frame of the first window triggering: ("hans", "el

Re: Trying to comprehend rolling windows + event time

I should add i'm using version 0.10.1 -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Trying-to-comprehend-rolling-windows-event-time-tp5034p5035.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: streaming hdfs sub folders

Hi Martin, where is the null pointer exception thrown? I think you didn't call the open() method of the AvroInputFormat. Maybe that's the issue. On Thu, Feb 18, 2016 at 5:01 PM, Martin Neumann wrote: > I tried to implement your idea but I'm getting NullPointer exceptions from > the AvroInputFor

Re: Trying to comprehend rolling windows + event time

Hi, in the second case: do you know what watermark is emitted after ("grace", "arctic", 25) is emitted? I imagine it is Long.MAX_VALUE, since otherwise all of the windows would not have been triggered. If there are no intermittent watermarks but we directly jump to the last (Long.MAX_VALUE) wate

Flink Stream assign unique transaction ID

Hi, I am currently working on Flink streaming API and I am trying to assign unique IDs to transactions to track changes and mine Frequent Item Sets. I identified several potential solutions such as: - Check pointing Local variables (snapshotState and restoreState for once only passage) - Key

Re: Flink Stream assign unique transaction ID

Hi, OperatorState was discarded in favor of ValueState, which you can retrieve using RuntimeContext.getState(…). This provides the same functionality as OperatorState but is the more future proof interface. Cheers, Aljoscha > On 19 Feb 2016, at 15:38, Tanguy Racinet wrote: > > Hi, > > I am cu

RE:Flink HA

I have resolved my issues. It seems that Avro had difficulties with my POJO. I change the management of the null value and it works fine. But, there is a way to cancel the old jobGraph who are starving in restarting status, and to keep the last one to restart ? Other than cancel JobId manually ?

state.backend.fs.checkpointdir setting

Hi All, I have been experiencing an error stopping my HA standalone setup. The cluster startup just fine, but when i deploy an application to it, I got this exception: java.lang.Exception: Call to registerInputOutput() of invokable failed at org.apache.flink.runtime.taskmanager.Task.ru

Re: Finding the average temperature

Hello Aljoscha , My sincere apologies at the beginning, if I seem to repeat the same question, almost interminably. If it is frustrating you, I seek your patience but I really want to nail it down in mind. :-) The point about parallelizing is well taken. I understand why the stream should be bro

How to increase akka heartbeat?

Hi, I have a Flink client application that launches jobs to remote clusters. However I'm getting my jobs cancelled: "18:25:29,650 WARN akka.remote.ReliableDeliverySupervisor- Association with remote system [akka.tcp://flink@127.0.0.1:52929] has failed, address is now gated

Re: How to increase akka heartbeat?

Hi Saiph, are you sure that the jobs are cancelled because the client disconnects? For the different timeouts, check the configuration page: https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/config.html and search for "heartbeat". On Fri, Feb 19, 2016 at 8:04 PM, Saiph Kappa wr

Re: How to increase akka heartbeat?

I am not sure. For that particular machine I get messages like these: « myip:6123/user/jobmanager#291801197])) at akka://flink/user/$a from Actor[akka://flink/deadLetters]. ^[[34m[INFO]^[[0;39m o.a.f.r.c.JobClientActor- Connected to new JobManager akka.tcp://flink@myip:6123/user/jobmanager. ^

Re: How to increase akka heartbeat?

These were the parameters that I set btw: akka.watch.heartbeat.interval: 100 akka.transport.heartbeat.interval: 1000 On Fri, Feb 19, 2016 at 7:43 PM, Saiph Kappa wrote: > I am not sure. > > For that particular machine I get messages like these: > « > myip:6123/user/jobmanager#291801197])) at ak

Re: How to increase akka heartbeat?

Hi, can you maybe (if you want also private) send me the full logs of the jobmanager? The messages you've posted here are logged at DEBUG level. They don't indicate an erroneous behavior of the system. On Fri, Feb 19, 2016 at 8:44 PM, Saiph Kappa wrote: > These were the parameters that I set btw

Re: How to increase akka heartbeat?

Hi Saiph! What is the problem that is happening? The log actually looks like the Job is successfully sent to the JobManager. Stephan On Fri, Feb 19, 2016 at 8:49 PM, Robert Metzger wrote: > Hi, > can you maybe (if you want also private) send me the full logs of the > jobmanager? The messages

Using numberOfTaskSlots to control parallelism

What would the differences be between these scenarios? 1) one task manager with numberOfTaskSlots=1 and one job with parallelism=1 2) one task manager with numberOfTaskSlots=10 and one job with parallelism=10 In both cases all of the job's tasks get executed within the one task manager's jvm. Ar

Trying to comprehend rolling windows + event time

Hello lofifnc I am keen to hear more about this particular thread of discussion. However, just a silly question: in the first case, why do you say that 'Each 5 times, as expected'! What causes them to appear 5 times? I don't see any _repeat()_ or _repeatAll()_ in the gist you have shared. What a