Here is my understanding of how job submission works in Flink. When
submitting a job to job manager via REST API, we provide a entry class. Job
manager then evaluate job graph and ship serialized operators to task
manager. Task manager then open operators and run tasks.
My app would typically requ
Hi Kien,
Thanks for the feedback. I wasn't certain regarding compatibility between
jars. I did version bump the flink libraries and the application did start.
Just curious if the previous jar still worked without upgrading.
Regarding the savepoint table. Someone should probably add 1.4 informatio
I recently tried to launch our application 1.3 jars against a 1.4 cluster.
I got a java.lang.NoClassDefFoundError:
org/apache/flink/streaming/api/checkpoint/CheckpointedRestoring when I
tried to run our 1.3 flink application against 1.4 .
Then I googled around and didn't see a mention of 1.4 in th
Hi Vishal,
AFAIK, intermittent restore failures from savepoints should not be expected.
Do you still have the logs from the failed restore attempts? What exceptions
were the restores failing on?
We would need to take a look at the logs to figure what may be going on.
Best,
Gordon
--
Sent from:
Hi Mans,
What's the difference between an operator and a function ?
An operator in Flink needs to handle processing of watermarks, records, and
checkpointing of the operator state.
To implement one, you need to extend the AbstractStreamOperator base class.
It is considered a very low-level API
And that further begs the question.. how performant is Timer Service. I
tried to peruse through the architecture behind it but cold not find a
definite clue. Is it a Scheduled Service and if yes how many threads etc...
On Wed, Dec 20, 2017 at 4:25 PM, Vishal Santoshi
wrote:
> Makes sense. Did a
Are intermittent failures to restore from a SP, in the case of flink
offset a know issue ? I had more than one instance where the offsets were
not restored, but a retry ( in one case I it succeeded like the 4th restore
attempt ) . I am on 1.3.2.
Makes sense. Did a first stab at Using ProcessFunction. The TimeService
exposed by the Context does not have remove timer. Is it primarily b'coz A
Priority Queue is the storage ad remove from a PriorityQueue is expensive
? Trigger Context does expose another version that has removal abilities
so w
Hi:
I am reading the documentation on working with state
(https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html)
and it states that :
All datastream functions can use managed state, but the raw state interfaces
can only be used when implementing operators. Using
Hi Vipul,
Thanks for the information. Yes, I do have checkpointing enabled with 10
millisecs.
I think the issue here is that the stream ended before the checkpoint
reached. This is a testing code that the DataStream only have 5 events then
it ended. Once the stream ended, the checkpoint is no
The prometheus metric reporter allows for a specification of a port range; is
there a way I can find out which actual port it found to bind to?
Also, there doesn’t seem to be a way to reserve an extra port for task managers
in mesos to assign to a metric reporter, is that a roadmap item? I’m abl
Libraries such as CEP or Table API should have the "compile" scope and
should be in the both the fat and non-fat jar.
The non-fat jar should contain everything that is not in flink-dist or
your lib directory.
Regards,
Timo
Am 12/20/17 um 3:07 PM schrieb shashank agarwal:
Hi,
In that case,
Hi,
In that case, it won't find the dependencies. Cause I have other
dependencies also and what about CEP etc. cause that is not part of
flink-dist.
Best
Shashank
On Wed, Dec 20, 2017 at 3:16 PM, Aljoscha Krettek
wrote:
> Hi,
>
> That jar file looks like it has too much stuff in there tha
Hi Colin,
the log looks as if the Flink JobManager receives a SIGTERM signal and
shuts down due to that. This is nothing which should be triggered by
Flink's leader election. Could you check whether this signal might be
created by another process in your environment or if the container
supervisor
Hi,
I think the cause is very likely a race condition between the tasks checking
and setting the static value, because tasks run in different threads. You could
try to use an Atomic reference or synchronization for setting the state
variable’s value.
Best,
Stefan
> Am 20.12.2017 um 00:29 schr
It turn out that our flink branch is out-of-date. Sorry for all the noise. :)
Regards,
Kien
Sent from TypeApp
On Dec 20, 2017, 16:42, at 16:42, Kien Truong wrote:
>Upon further investigation, we found out that the reason:
>
>* The cluster was started on YARN with the hadoop classpath, which
Hi,
That jar file looks like it has too much stuff in there that shouldn't be
there. This can explain the errors you seeing because of classloading conflicts.
Could you try not building a fat-jar and have only your code in your jar?
Best,
Aljoscha
> On 20. Dec 2017, at 10:15, shashank agarwal
Upon further investigation, we found out that the reason:
* The cluster was started on YARN with the hadoop classpath, which includes
Avro. Therefore, Avro's SpecificRecord class was loaded using
sun.misc.Launcher$AppClassLoader
* Our LteSession class was submitted with the application jar, an
One more thing when i submit the job ir start yarn session it prints
following logs :
Using the result of 'hadoop classpath' to augment the Hadoop classpath:
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6
Hey Jinhua,
- The `slaves` file is only relevant for the startup scripts. You can
add as many task managers as you like by starting them manually.
- You can check the logs of the JobManager or its web UI
(jobmanager-host:8081) to see how many TMs have registered.
- The registration time out looks
Hi,
Could you please list what exactly is in your submitted jar file, for example
using "jar tf my-jar-file.jar"? And also what files exactly are in your Flink
lib directory.
Best,
Aljoscha
> On 19. Dec 2017, at 20:10, shashank agarwal wrote:
>
> Hi Timo,
>
> I am using Rocksdbstatebackend
Would it be possible to get the same result using windows?
Jayant Ameta
On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz <
wysakowicz.da...@gmail.com> wrote:
> It is not possible at this moment. FlinkCEP can handle only one Pattern
> applied statically. There is a JIRA ticket for that:
> https:
23 matches
Mail list logo
