thanks for the response, with respect to the api-server i don't think i can do so much about it because i am just using a specific namespace in kubernetes cluster, it's not me who administrate the cluster.
otherwise i will try the gc log option to see if can find something useful in order to debug this problem. Le jeu. 9 sept. 2021 à 22:25, houssem <mejrihousse...@gmail.com> a écrit : > Hello , > > with respect to the api-server i dotn re > > On 2021/09/09 11:37:49, Yang Wang <danrtsey...@gmail.com> wrote: > > I think @Robert Metzger <rmetz...@apache.org> is right. You need to > check > > whether your Kubernetes APIServer is working properly or not(e.g. > > overloaded). > > > > Another hint is about the fullGC. Please use the following config option > to > > enable the GC logs and check the full gc time. > > env.java.opts.jobmanager: -verbose:gc -XX:+PrintGCDetails > > -XX:+PrintGCDateStamps -Xloggc:/opt/flink/log/jobmanager-gc.log > > > > Simply increasing the renew-deadline might help. But it could not solve > the > > problem completely. > > high-availability.kubernetes.leader-election.lease-duration: 120 s > > high-availability.kubernetes.leader-election.renew-deadline: 120 s > > > > > > Best, > > Yang > > > > Robert Metzger <rmetz...@apache.org> 于2021年9月9日周四 下午6:52写道： > > > > > Is the kubernetes server you are using particularly busy? Maybe these > > > issues occur because the server is overloaded? > > > > > > "Triggering checkpoint 2193 (type=CHECKPOINT) @ 1630681482667 for job > > > 00000000000000000000000000000000." > > > "Completed checkpoint 2193 for job 00000000000000000000000000000000 > (474 > > > bytes in 195 ms)." > > > "Triggering checkpoint 2194 (type=CHECKPOINT) @ 1630681492667 for job > > > 00000000000000000000000000000000." > > > "Completed checkpoint 2194 for job 00000000000000000000000000000000 > (474 > > > bytes in 161 ms)." > > > "Renew deadline reached after 60 seconds while renewing lock > > > ConfigMapLock: myNs - myJob-dispatcher-leader > > > (1bcda6b0-8a5a-4969-b9e4-2257c4478572)" > > > "Stopping SessionDispatcherLeaderProcess." > > > > > > At some point, the leader election mechanism in fabric8 seems to give > up. > > > > > > > > > On Tue, Sep 7, 2021 at 10:05 AM mejri houssem < > mejrihousse...@gmail.com> > > > wrote: > > > > > >> hello, > > >> > > >> Here's other logs of the latest jm crash. > > >> > > >> > > >> Le lun. 6 sept. 2021 à 14:18, houssem <mejrihousse...@gmail.com> a > > >> écrit : > > >> > > >>> hello, > > >>> > > >>> I have three jobs running on my kubernetes cluster and each job has > his > > >>> own cluster id. > > >>> > > >>> On 2021/09/06 03:28:10, Yangze Guo <karma...@gmail.com> wrote: > > >>> > Hi, > > >>> > > > >>> > The root cause is not "java.lang.NoClassDefFound". The job has been > > >>> > running but could not edit the config map > > >>> > "myJob-00000000000000000000000000000000-jobmanager-leader" and it > > >>> > seems finally disconnected with the API server. Is there another > job > > >>> > with the same cluster id (myJob) ? > > >>> > > > >>> > I would also pull Yang Wang. > > >>> > > > >>> > Best, > > >>> > Yangze Guo > > >>> > > > >>> > On Mon, Sep 6, 2021 at 10:10 AM Caizhi Weng <tsreape...@gmail.com> > > >>> wrote: > > >>> > > > > >>> > > Hi! > > >>> > > > > >>> > > There is a message saying "java.lang.NoClassDefFound Error: > > >>> org/apache/hadoop/hdfs/HdfsConfiguration" in your log file. Are you > > >>> visiting HDFS in your job? If yes it seems that your Flink > distribution or > > >>> your cluster is lacking hadoop classes. Please make sure that there > are > > >>> hadoop jars in the lib directory of Flink, or your cluster has set > the > > >>> HADOOP_CLASSPATH environment variable. > > >>> > > > > >>> > > mejri houssem <mejrihousse...@gmail.com> 于2021年9月4日周六 上午12:15写道： > > >>> > >> > > >>> > >> > > >>> > >> Hello , > > >>> > >> > > >>> > >> I am facing a JM crash lately. I am deploying a flink > application > > >>> cluster on kubernetes. > > >>> > >> > > >>> > >> When i install my chart using helm everything works fine but > after > > >>> some time ,the Jm starts to crash > > >>> > >> > > >>> > >> and then it gets deleted eventually after 5 restarts. > > >>> > >> > > >>> > >> flink version: 1.12.5 (upgraded recently from 1.12.2) > > >>> > >> HA mode : k8s > > >>> > >> > > >>> > >> Here's the full log of the JM attached file. > > >>> > > > >>> > > >> > > >
