additionally we have these coming with this as well all the time:
com.esotericsoftware.kryo.KryoException:
java.lang.ArrayIndexOutOfBoundsException
Serialization trace:
_children (com.fasterxml.jackson.databind.node.ObjectNode)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(Obj
Hi, hu
I am not sure why do you need to start multiple jobmanagers on kubernetes.
Just as the manual [1], we use a deployment of 1 to make sure kubernetes
detect the crash of jobmanager and start a new one. What we should do is to
add the high availability configurations [2] in flink-conf.yaml. Yo
So here's the thing: Metrics are accurate, so long as the job is
running. Once the job terminates metrics are cleaned up and not
persisted anywhere, with the exception of a few metrics (like numRecordsIn).
Another thing that is always good to double-check is to enable DEBUG
logging and re-run
Hi Leo,
Recently, I came across kinesis data analytics which provides managed flink
in AWS. It seems promising.
Some concerns that I have with my very little exploration till now are -
=> it uses flink 1.6 (not the latest one)
=> Doesn't support kafka streams directly
=> No option to configure ta
Hi, Vadim
This similar issue has occurred in earlier versions, see
https://issues.apache.org/jira/browse/FLINK-4485.
Are you running a Flink session cluster on YARN? I think it might be a bug.
I'll see if I can reproduce it with the master branch code, and if yes, I will
try to investigate it.
HI Tan:
I have the same problem with you when running "flink-1.7.2 ON KUBERNATE HA"
mode, may I ask if you have solved this problem? How? After I started the two
jobmanagers normally, when I tried to kill one of them, he could not restart
normally. Both jobmanagers reported this error. The speci
Hi, Andrea
If you are running flink cluster on Yarn, the jar
`flink-shaded-hadoop2-uber-1.6.4.jar` should exist in the lib dir of the
flink client, so that it could be uploaded to the Yarn Distributed Cache
and then be available on JM and TM.
And if you are running flink standalone cluster, the j
Hi Xintong: Thanks for your pointers.
I tried -Dparallelism.default=2 locally (in IDE) and it did not work. Do you
know if there is a common way that would work both for emr, locally and ide ?
Thanks again.
On Thursday, June 27, 2019, 12:03:06 AM EDT, Xintong Song
wrote:
Hi Singh,
Yo
Hi Chesnay:
Thanks for your response.
My job runs for a few minutes and i've tried setting the reporter interval to 1
second.
I will try the counter on a longer running job.
Thanks again.
On Thursday, June 27, 2019, 11:46:17 AM EDT, Chesnay Schepler
wrote:
1) None that I'm aware of.
Hi guys,
I have a simple batch job with a custom output formatter that writes to a local
file.
public class JobHadoop {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
env.fromCollection(
Hi, we have some Flink Jobs (Flink Version 1.7.1) consuming from a Custom
Source and Ingesting into an Elasticsearch Cluster (V.5.6). In recent
times, we see more and more Exceptions happening like this:
com.esotericsoftware.kryo.KryoException: Unable to find class: com. ^
at
com.esoteric
1) None that I'm aware of.
2) You should use counters.
3) No, counters are not checkpointed, but you could store the value in
state yourself.
4) None that I'm aware of that doesn't require modifications to the
application logic.
How long does your job run for, and how do you access metrics?
O
Hi:
I need to collect application metrics which are counts (per unit of time eg:
minute) for certain events. There are two ways of doing this:
1. Create separate streams (using split stream etc) in the application
explicitly, then aggregate the counts in a window and save them. This mixes
met
Hi Haibo Sun,
It's exactly what I needed. Thank you so much!
Best,
Pankaj
On Thu, Jun 27, 2019 at 7:45 AM Haibo Sun wrote:
> Hi, Pankaj Chand
>
> If you're running Flink on YARN, you can do this by limiting the number of
> applications in the cluster or in the queue. As far as I know, Flink
Hi, Pankaj Chand
If you're running Flink on YARN, you can do this by limiting the number of
applications in the cluster or in the queue. As far as I know, Flink does not
limit that.
The following are the configuration items for YARN :
yarn.scheduler.capacity.maximum-applications
ya
Hi everyone,
Is there any way (parameter or function) I can limit the number of
concurrent jobs executing in my Flink cluster? Or alternatively, limit the
number of concurrent Job Managers (since there has to be one Job Manager
for every job)?
Thanks!
Pankaj
HI Qiu,
my jar does not contain the class
`org.apache.hadoop.hdfs.protocol.HdfsConstants*`, *but I do expect it is
contained within `flink-shaded-hadoop2-uber-1.6.4.jar` which is located in
Flink cluster libs.
Il giorno gio 27 giu 2019 alle ore 04:03 Congxian Qiu <
qcx978132...@gmail.com> ha scrit
Got it, it's clear, TableStats is the important functions of ExternalCatalog.
It is right way.
Best, JingsongLee
--
From:Felipe Gutierrez
Send Time:2019年6月27日(星期四) 14:53
To:JingsongLee
Cc:user
Subject:Re: Hello-world example of
18 matches
Mail list logo
