Actually, I was wrong. It turns out I was setting the values the wrong way in conf/flink-conf.yaml. I set "metrics.latency.interval 100" instead of "metrics.latency.interval: 100". Sorry about that.
T On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand <pankajchanda...@gmail.com> wrote: > Thank you, David! > > After setting ExecutionConfig for latency tracking in the > SocketWindowWordCount Java source code and rebuilding and using that > application jar file, I am now getting the latency tracking metrics using > the REST endpoint. The below documentation [1] seems to imply that > merely setting the parameters in flink-conf.yaml would generate the latency > tracking metrics, which is not happening in my case. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking > > On Wed, Sep 9, 2020 at 10:08 AM David Anderson <da...@alpinegizmo.com> > wrote: > >> Pankaj, >> >> I just checked, and the latency metrics for SocketWindowWordCount show up >> just fine for me with Flink 1.11.1. For me, the latency metrics existed >> even before I provided any data on the socket for the job to process. This >> makes sense, as the latency tracking markers will propagate through the job >> graph and provide measurements despite the lack of any real data for the >> job to process. >> >> These metrics have horrible names, but for me, >> >> curl -s >> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics >> >> reveals a bunch of latency metrics, and, for example, >> >> curl -s >> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999 >> >> returns >> >> >> [{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}] >> >> Best, >> David >> >> >> On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <pankajchanda...@gmail.com> >> wrote: >> >>> Hi David, >>> >>> Thanks for replying! Sorry, I forgot to mention I am using >>> Flink Version: 1.11.1, Commit ID: 7eb514a. >>> Is it possible that the default SocketWindowWordCount job is too simple >>> to generate Latency metrics? Or that the latency metrics disappear from the >>> output JSON when the data ingestion is zero? >>> >>> Thanks, >>> >>> Pankaj >>> >>> >>> On Wed, Sep 9, 2020 at 6:27 AM David Anderson <da...@alpinegizmo.com> >>> wrote: >>> >>>> Pankaj, >>>> >>>> The Flink web UI doesn't do any visualizations of histogram metrics, so >>>> the only way to access the latency metrics is either through the REST api >>>> or a metrics reporter. >>>> >>>> The REST endpoint you tried is the correct place to find these metrics >>>> in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 >>>> (if I recall correctly) these metrics were task metrics. So if you are >>>> using an older version of Flink you'll need to dig deeper. I believe you'll >>>> find them in >>>> >>>> /jobs/<jobid>/vertices/<vertexid>/subtasks/metrics >>>> >>>> Regards, >>>> David >>>> >>>> >>>> >>>> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <pankajchanda...@gmail.com> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> How do I visualize (or extract) the results for Latency Tracking for a >>>>> Flink local cluster? I set "metrics.latency.interval 100" in the >>>>> conf/flink-conf.yaml file, and started the cluster and >>>>> SocketWindowWordCount job. However, I could not find the latency >>>>> distributions anywhere in the web UI, nor are there any latency metrics in >>>>> the Metrics dropdown box for either task. >>>>> >>>>> I also set "metrics.latency.granularity "operator"" in >>>>> conf/flink-conf.yaml, but that did not help. >>>>> >>>>> When I tried to query the REST endpoint, I got the following output >>>>> which did not seem to contain anything related to latency: >>>>> >>>>> $ curl -s >>>>> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics >>>>> >>>>> >>>>> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}] >>>>> >>>>> Thanks, >>>>> >>>>> Pankaj >>>>> >>>>
