Hi Team,
How can we configure Flink History Server to retrieve the logs from
jobManager and taskManagers?
Currently, all of our flink logs are getting stored in ElasticSearch but we
want to observe these logs from History Server as well.
Any sort of suggestions would be very helpful.
Thanks
uot;job-accumulators\":[],\"user-task-accumulators\":[{\"name\":\"processed\",\"type\":\"IntCounter\",\"value\":\"86082\"},{\"name\":\"max_processed_per_key\",\"type\":\"IntMaximum\",
g/flink/flink-docs-release-1.7/api/java/index.html?org/apache/flink/api/common/accumulators/class-use/Accumulator.html
>> >
>> > I expected the job to display the accumulator results in the Flink History
>> > Server after completion, but the UI remains blank. After some
d using the
> Accumulator API to develop some statistics:
> https://nightlies.apache.org/flink/flink-docs-release-1.7/api/java/index.html?org/apache/flink/api/common/accumulators/class-use/Accumulator.html
> >
> > I expected the job to display the accumulator results in the Flink
> His
de and using the Accumulator
> API to develop some statistics:
> https://nightlies.apache.org/flink/flink-docs-release-1.7/api/java/index.html?org/apache/flink/api/common/accumulators/class-use/Accumulator.html
>
> I expected the job to display the accumulator results in the Flink Histo
accumulator results in the Flink History
Server after completion, but the UI remains blank. After some
investigation, I discovered that the accumulator values are only visible
while the job is running.
Is this the expected behavior? Does the community have any plans to enable
viewing accumulator results
Thanks for sharing your thoughts, Gyula!
>Not really sure what you are trying to achieve with it , maybe some other
audit feature would be enough to simply track the spec changes over time of
the CR?
Basically we wanted to leverage Flink history server to know the details /
insights(e.g.
;last-state" so that's not a big problem.
Cancelling during the last-state upgrade instead of using the HA metadata
will generate a new job.id but it also be a slightly slower process overall.
To be honest I don't know many people who are using the history server like
this for str
n operator main). I am actually working on adding a new
> way to perform the last-state upgrade via simple cancellation but that's a
> slightly orthogonal question.
>
> Long story short if you really need to integrate this with the history
> server, then you should switch to sa
am actually working on adding a new
way to perform the last-state upgrade via simple cancellation but that's a
slightly orthogonal question.
Long story short if you really need to integrate this with the history
server, then you should switch to savepoint upgrades.
Cheers,
Gyula
On Wed, A
eployments / upgrades, because the operator uses the
job failover mechanism to achieve "last-state" upgrade mode.
However, with it, it seems impossible to integrate with Flink history
server which uses job.id to differentiate different job deployments.
Questions:
- Is there any wa
In Flink k8s application mode with high-availability, it's job id always
00, but in history server, it make job's id for the key. How can I
using the application mode with HA and store the history job status with
history server?
Best,
tanjialiang.
- -c
>>>>> - /opt/flink/bin/flink run-application --target
>>>>> kubernetes-application -Dkubernetes.service-account=flink-service-account
>>>>> -Dkubernetes.rest-service.exposed.type=NodePort
>&g
ion -Dkubernetes.service-account=flink-service-account
>>>> -Dkubernetes.rest-service.exposed.type=NodePort
>>>> -Dkubernetes.cluster-id=batch-job-cluster
>>>> -Dkubernetes.container.image=localhost:5000/batch-flink-app-v3:latest
>>>> -Ds3.endpoint=http
app-v3:latest
>>> -Ds3.endpoint=http://minio-1616518256:9000 -Ds3.access-key=ACCESSKEY
>>> -Ds3.secret-key=SECRETKEY
>>> -Djobmanager.archive.fs.dir=s3://flink/completed-jobs/
>>> -Ds3.path-style-access=true -Ds3.ssl.enabled=false
>>> -Dhigh-availability=org.apache.flink.kuber
-Ds3.secret-key=SECRETKEY
>> -Djobmanager.archive.fs.dir=s3://flink/completed-jobs/
>> -Ds3.path-style-access=true -Ds3.ssl.enabled=false
>> -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>> -Dhigh-availability.storageDir=s3://flink/flink-ha
>&g
abled=false
> -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> -Dhigh-availability.storageDir=s3://flink/flink-ha
> local:///opt/flink/usrlib/job.jar
> restartPolicy: OnFailure
>
>
> This works well for me but I would like to write
ory
-Dhigh-availability.storageDir=s3://flink/flink-ha
local:///opt/flink/usrlib/job.jar
restartPolicy: OnFailure
This works well for me but I would like to write the result to the archive
path and show it in the History server (running as separate deployment in
k8)
Anytime it cre
Thank you for the confirmation.
On Fri, Mar 19, 2021 at 5:37 AM Matthias Pohl
wrote:
> Hi Vishal,
> yes, as the documentation explains [1]: Only jobs that reached a globally
> terminal state are archived into Flink's history server. State information
> about running jobs
Hi Vishal,
yes, as the documentation explains [1]: Only jobs that reached a globally
terminal state are archived into Flink's history server. State information
about running jobs can be retrieved through Flink's REST API.
Best,
Matthias
[1]
https://ci.apache.org/projects/flink/flink-do
Hello folks,
Does fliink server not provide for running jobs ( like spark history does )
?
Regards.
’t do this?
2020-07-11 11:43:29,527 [HistoryServer shutdown hook] INFO
HistoryServer - *Removing web dashboard root cache directory
/local/scratch/flink_historyserver_tmpdir*
2020-07-11 11:43:29,536 [HistoryServer shutdown hook] INFO
HistoryServer - Stopped history server.
We’re attempting to w
[HistoryServer shutdown hook] INFO HistoryServer -
Stopped history server.
We're attempting to work around the UI becoming un-responsive/crashing the
browser at a large number archives (in my testing, that's around 20,000
archives with Chrome) by persisting the job IDs of our submitted app
at is the upper limit of the number of archives the history server
can support? Does it attempt to download every archive and load them
all into memory?
2.Retention: we have on the order of 100K applications per day in our
production environment. Is there any native retention of policy? E.g.
f you please:
1. What is the upper limit of the number of archives the history server
can support? Does it attempt to download every archive and load them all into
memory?
2. Retention: we have on the order of 100K applications per day in our
production environment. Is there any n
us to know if we're encountering some kind
of resource problem.
// ah
From: Hailu, Andreas [Engineering]
Sent: Thursday, May 28, 2020 12:18 PM
To: 'Chesnay Schepler' <mailto:ches...@apache.org>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: RE: History
lem.
*// *ah**
*From:*Hailu, Andreas [Engineering]
*Sent:* Thursday, May 28, 2020 12:18 PM
*To:* 'Chesnay Schepler' ; user@flink.apache.org
*Subject:* RE: History Server Not Showing Any Jobs - File Not Found?
Okay, I will look further to see if we’re mistakenly using a version
that
Andreas [Engineering]
Sent: Thursday, May 28, 2020 12:18 PM
To: 'Chesnay Schepler' ; user@flink.apache.org
Subject: RE: History Server Not Showing Any Jobs - File Not Found?
Okay, I will look further to see if we're mistakenly using a version that's
pre-2.6.0. However, I don't
4j12-1.7.15.jar
Are the files within /lib.
// ah
From: Chesnay Schepler
Sent: Thursday, May 28, 2020 11:00 AM
To: Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
Looks like it is indeed stuck on downloading the archive.
I
jar being included introduce?
*// *ah**
*From:*Chesnay Schepler
*Sent:* Thursday, May 28, 2020 9:26 AM
*To:* Hailu, Andreas [Engineering] ;
user@flink.apache.org
*Subject:* Re: History Server Not Showing Any Jobs - File Not Found?
If it were a class-loading issue I would think that we'd
ecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
What problems could the flink-shaded-hadoop jar being included introduce?
// ah
From: Chesnay Schepler
Sent: Thursday, May 28, 2020 9:26 AM
To: Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject:
ngineering] ;
user@flink.apache.org
*Subject:* Re: History Server Not Showing Any Jobs - File Not Found?
yes, exactly; I want to rule out that (somehow) HDFS is the problem.
I couldn't reproduce the issue locally myself so far.
On 01/05/2020 22:31, Hailu, Andreas wrote:
Hi Chesnay,
Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
yes, exactly; I want to rule out that (somehow) HDFS is the problem.
I couldn't reproduce the issue locally myself so far.
On 01/05/2020 22:31, Hailu, Andreas wrote:
Hi Ch
user@flink.apache.org
*Subject:* Re: History Server Not Showing Any Jobs - File Not Found?
hmm...let's see if I can reproduce the issue locally.
Are the archives from the same version the history server runs on?
(Which I supposed would be 1.9.1?)
Just for the sake of narrowing things down, it wou
ctory?
// ah
From: Chesnay Schepler
Sent: Wednesday, April 29, 2020 8:26 AM
To: Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
hmm...let's see if I can reproduce the issue locally.
Are the archives from the same version t
hmm...let's see if I can reproduce the issue locally.
Are the archives from the same version the history server runs on?
(Which I supposed would be 1.9.1?)
Just for the sake of narrowing things down, it would also be interesting
to check if it works with the archives residing in the
l 27, 2020 10:28 AM
To: Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
If historyserver.web.tmpdir is not set then java.io.tmpdir is used, so that
should be fine.
What are the contents of /local/scratch/flink_historyserver
/
historyserver.web.tmpdir: /local/scratch/flink_historyserver_tmpdir/
Did you have anything else in mind when you said pointing somewhere funny?
*// *ah**
*From:*Chesnay Schepler
*Sent:* Monday, April 27, 2020 5:56 AM
*To:* Hailu, Andreas [Engineering] ;
user@flink.apache.org
*Subject:* Re: History Server Not
k.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
overview.json is a generated file that is placed in the local directory
controlled by historyserver.web.tmpdir.
Have you configured this option to point to some non-local filesystem? (Or if
not, is the java.io.tmpd
u, Andreas
[Engineering] ; user@flink.apache.org
*Subject:* RE: History Server Not Showing Any Jobs - File Not Found?
Hi Chesnay, thanks for responding. We’re using Flink 1.9.1. I enabled
DEBUG level logging and this is something relevant I see:
2020-04-22 13:25:52,566 [Flink-HistoryServer-Arch
as [Engineering]
; user@flink.apache.org
Subject: RE: History Server Not Showing Any Jobs - File Not Found?
Hi Chesnay, thanks for responding. We're using Flink 1.9.1. I enabled DEBUG
level logging and this is something relevant I see:
2020-04-22 13:25:52,566 [Flink-HistoryServer-ArchiveFet
esday, April 22, 2020 2:16 AM
To: Hailu, Andreas [Engineering] ;
user@flink.apache.org
Subject: Re: History Server Not Showing Any Jobs - File Not Found?
Which Flink version are you using?
Have you checked the history server logs after enabling debug logging?
On 21/04/2020 17:16, Hailu, An
Which Flink version are you using?
Have you checked the history server logs after enabling debug logging?
On 21/04/2020 17:16, Hailu, Andreas [Engineering] wrote:
Hi,
I’m trying to set up the History Server, but none of my applications
are showing up in the Web UI. Looking at the console, I
Hi,
I'm trying to set up the History Server, but none of my applications are
showing up in the Web UI. Looking at the console, I see that all of the calls
to /overview return the following 404 response: {"errors":["File not found."]}.
I've set up my configu
Hi pwestermann
I believe this is related to
https://issues.apache.org/jira/browse/FLINK-13799
It seems that the configuration.features['web-submit'] is missed from the
api when you upgrading from 1.7 to 1.9.2
Do you have the same problem when upgrading to 1.10? feel free to ping me if
you still
Hey Robert,
I just tried Flink 1.10 and the history server UI works for me too. Only
Flink 1.9.2 is not loading.
Since we were already looking into upgrading to 1.10, I might just do that
now.
Thanks,
Peter
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hey Peter,
I tried reproducing the error, and for a second, I though the 1.10 release
really broke the web ui, because I saw a pretty similar error.
However after clearing the cache, the error was gone.
Are you sure that you cleared the cache of your browser?
I have also asked the main contributo
I am seeing this error in firefox:
ERROR TypeError: "this.statusService.configuration.features is undefined"
t http://10.25.197.60:8082/main.177039bdbab11da4f8ac.js:1
qr http://10.25.197.60:8082/main.177039bdbab11da4f8ac.js:1
Gr http://10.25.197.60:8082/main.177039bdbab11da4f8ac.js:1
fresh(e.g. Cmd+Shft+R for Mac). It solved my
> problem before.
>
>
> Best,
> Yang
>
> pwestermann 于2020年3月4日周三 下午8:40写道：
>
>> We recently upgraded from Flink 1.7 to Flink 1.9.2 and the history server
>> UI
>> now seems to be broken. It doesn't load
If all the rest api could be viewed successfully, then the reason may be js
cache.
You could try to force a refresh(e.g. Cmd+Shft+R for Mac). It solved my
problem before.
Best,
Yang
pwestermann 于2020年3月4日周三 下午8:40写道：
> We recently upgraded from Flink 1.7 to Flink 1.9.2 and the history ser
We recently upgraded from Flink 1.7 to Flink 1.9.2 and the history server UI
now seems to be broken. It doesn't load and always just displays a blank
screen.
The individual endpoints (e.g. /jobs/overview) still work.
Could this be an issue caused by the Angular update for the regul
2020-02-18 09:44:45,227 ERROR
org.apache.flink.runtime.webmonitor.hist/ry.HistoryServerArchiveFetcher -
Failure while fetching/process
ing job archive for job eaf0639027aca1624adaa100bdf1332e.
java.nio.file.FileSystemException:
/dev/shm/flink-history-server/jobs/eaf0639027aca1624adaa100bdf1332e
rtition were full or not?
>>>
>>> Richard Moorhead 于2020年2月18日周二 上午8:16写道：
>>>
>>>> I see the following exception often:
>>>>
>>>> 2020-02-17 18:13:26,796 ERROR
>>>> org.apache.flink.runtime.webmonitor.history.HistoryServer
ime.webmonitor.history.HistoryServerArchiveFetcher -
>>> Failure while fetching/processing job archive for job
>>> eaf0639027aca1624adaa100bdf1332e.
>>> java.nio.file.FileSystemException:
>>> /dev/shm/flink-history-server/jobs/eaf0639027aca1624adaa100bdf1332e/vertices/6a
t;> 2020-02-17 18:13:26,796 ERROR
>> org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher -
>> Failure while fetching/processing job archive for job
>> eaf0639027aca1624adaa100bdf1332e.
>> java.nio.file.FileSystemException:
>> /dev/shm/flink-history-server/jobs/eaf06390
; Failure while fetching/processing job archive for job
> eaf0639027aca1624adaa100bdf1332e.
> java.nio.file.FileSystemException:
> /dev/shm/flink-history-server/jobs/eaf0639027aca1624adaa100bdf1332e/vertices/6abf3ed37d1a5e48f2786b832033f074/subtasks/86/attempts:
> No space
I see the following exception often:
2020-02-17 18:13:26,796 ERROR
org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher -
Failure while fetching/processing job archive for job
eaf0639027aca1624adaa100bdf1332e.
java.nio.file.FileSystemException:
/dev/shm/flink-history-server
I think the best way to view the log is flink history server.
However, it could only support jobGraph and exceptions. Maybe
the flink history server needs to be enhanced so that we could view
logs just like the cluster is running.
Best,
Yang
Yu Yang 于2019年9月6日周五 上午3:06写道：
> Hi Yun Tang &a
y will not be removed.
>
> Best
> Yun Tang
>
> --
> *From:* Zhu Zhu
> *Sent:* Friday, August 30, 2019 16:24
> *To:* Yu Yang
> *Cc:* user
> *Subject:* Re: best practices on getting flink job logs from Hadoop
> history server?
>
> Hi Yu,
>
> Re
note that the temporary files of the YARN session in the home
directory will not be removed.
Best
Yun Tang
From: Zhu Zhu
Sent: Friday, August 30, 2019 16:24
To: Yu Yang
Cc: user
Subject: Re: best practices on getting flink job logs from Hadoop history
server?
Hi
Hi Yu,
Regarding #2,
Currently we search task deployment log in JM log, which contains info of
the container and machine the task deploys to.
Regarding #3,
You can find the application logs aggregated by machines on DFS, this path
of which relies on your YARN config.
Each log may still include mu
Hi,
We run flink jobs through yarn on hadoop clusters. One challenge that we
are facing is to simplify flink job log access.
The flink job logs can be accessible using "yarn logs $application_id".
That approach has a few limitations:
1. It is not straightforward to find yarn application id ba
Hi Encho,
currently, the existing image does not support to start a HistoryServer.
The reason is simply that it has not been exposed because the image
contains everything needed. In order to do this, you would need to extend
the docker-entrypoint.sh script with an additional history-server option
Hello,
I am struggling to find how to run a history server in Kubernetes. The
docker image takes an argument that starts a jobmanager or a taskmanager,
but no history server. What's the best way to set up one in K8S?
Thanks,
Encho
The HS isn't aware of running jobs at all, so yes users would have to
switch once the job is done.
On 17.01.2018 00:45, Eron Wright wrote:
As a follow-up question, how well does the history server work for
observing a running job? I'm trying to understand whether, in the
clust
As a follow-up question, how well does the history server work for
observing a running job? I'm trying to understand whether, in the
cluster-per-job model, a user would be expected to hop from the Web UI to
the History Server once the job completed.
Thanks
On Wed, Oct 4, 2017 at 3:
To add to this:
The History Server is mainly useful in cases where one runs a
Flink-cluster-per-job. One the job finished, the processes disappear. The
History Server should be longer lived to make past executions' stats
available.
On Mon, Sep 25, 2017 at 3:44 PM, Nico Kruber wrote:
>
Hi Elias,
in theory, it could be integrated into a single web interface, but this was
not done so far.
I guess the main reason for keeping it separate was probably to have a better
separation of concerns as the history server is actually independent of the
current JobManager execution and
I am curious, why is the History Server a separate process and Web UI
instead of being part of the Web Dashboard within the Job Manager?
69 matches
Mail list logo
