Thanks for reporting this. I know there's problems with the plumbing of
metrics in the FlinkRunner, but I don't know the full extent. I'm actually
starting to work on that right now. I filed
https://github.com/apache/beam/issues/32895 with your report and I'll add
others as I gather them up.
Kenn
On Tue, Oct 22, 2024 at 12:43 AM Aditya Shirsath <adishirsa...@gmail.com>
wrote:
> Hi,
>
> I also tried to export all the metrics using flink configuration to the
> prometheus metric reporter. There I'm able to see only the metrics
> generated by Flink.
>
> On Tue, Oct 22, 2024 at 9:47 AM Ahmet Altay via user <user@beam.apache.org>
> wrote:
>
>> Adding Jan & Ken as people with potential knowledge in this area.
>>
>> On Sun, Oct 20, 2024 at 10:43 PM Aditya Shirsath <adishirsa...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm creating a beam pipeline to run on flink runner. I've created
>>> custom metrics in beam(JAVA SDK). But I'm unable to see it in flink
>>> dashboard or flink rest API response. Although I'm able to see under
>>> Accumulators section but not under metrics. Under accumulators I can see,
>>>
>>> { "metrics": { "attempted": [{ "urn": "beam:metric:user:sum_int64:v1",
>>> "type": "beam:metrics:sum_int64:v1", "payload": "FA==", "labels": {
>>> "NAMESPACE": "Bingo", "NAME": "bingotest", "PTRANSFORM":
>>> "log/ParMultiDo(LogOutput)" } }] } }
>>>
>>> I'm also able to log the metrics in beam part.
>>> I'm using Apache Beam 2.55.0 and Flink runner 1.16.
>>>
>>> Please let me know if this is the correct way to do this or am I missing
>>> something?
>>>
>>> Do I need to create a separate HTTP sink for this as mentioned here
>>> <https://beam.apache.org/documentation/programming-guide/#metrics> ?.
>>> But I couldn't find any other documentation to export data. Could you
>>> please help me here?
>>>
>>> Thank you!
>>>
>>
