Hi, after some investigation, here is the final findings:
1. The job accumulator, which represents 1 value accumulated across all
tasks, does show up in the job history file.
lniu@foo:~$ jq . d9830732259de77a04958f4f8ac40a76 | grep
user-task-accumulators
"json":
"{\"job-accumulators\":[],\"user-task-accumulators\":[{\"name\":\"processed\",\"type\":\"IntCounter\",\"value\":\"86082\"},{\"name\":\"max_processed_per_key\",\"type\":\"IntMaximum\",\"value\":\"2987\"}],\"serialized-user-task-accumulators\":......}"
2. API
http://history-server/jobs/d9830732259de77a04958f4f8ac40a76/accumulators
can be used to query the accumulator
3. the problem is there is no place in the history server to show the job
accumulator (at least I couldn't find it in the 1.18.1 history server).
The page I checked before was accumulator per subtask page. That is
expected to be empty in the history server because it need to query task
manager to get the value.
Best
Lu
On Tue, Apr 29, 2025 at 1:44 AM archzi lu <archzi...@gmail.com> wrote:
> Hi Lu Niu,
> Your scenario must be same with me.
> I'm so glad to share my solution with you.
> 1. use the operator state for save the custom statistics data,
> 2. save the data to external storage when the operator is closing.
> 3. query from external storage, and make some reduce operation with the
> data
>
> hope my solution can help you.
>
> Best.
> Jiadong Lu
>
> Lu Niu <qqib...@gmail.com> 于2025年4月29日周二 13:17写道：
> >
> > Thanks Jiadong!
> >
> > Could you elaborate more? I use application mode to run job in yarn, and
> the yarn application will shutdown right after the flink batch job completed
> >
> > Best
> > Lu
> >
> > On Mon, Apr 28, 2025 at 6:49 PM archzi lu <archzi...@gmail.com> wrote:
> >>
> >> hi, Lu Niu
> >>
> >> As a workaround, you could use the following ways to get the custom
> >> accumulator you defined.
> >> 1. RestCluserClient#getAccumulators method
> >> 2. /jobs/:jobid/accumulators Rest API
> >>
> >> Best.
> >> Jiadong Lu
> >>
> >> Lu Niu <qqib...@gmail.com> 于2025年4月29日周二 00:41写道：
> >> >
> >> > I am currently running a Flink job in batch mode and using the
> Accumulator API to develop some statistics:
> https://nightlies.apache.org/flink/flink-docs-release-1.7/api/java/index.html?org/apache/flink/api/common/accumulators/class-use/Accumulator.html
> >> >
> >> > I expected the job to display the accumulator results in the Flink
> History Server after completion, but the UI remains blank. After some
> investigation, I discovered that the accumulator values are only visible
> while the job is running.
> >> >
> >> > Is this the expected behavior? Does the community have any plans to
> enable viewing accumulator results after the job has completed, similar to
> how it works in Spark? Thank you!
> >> >
> >> > best
> >> > Lu
>
