Hi Pablo,
Thanks so much for your explanation!
Also, for prom client in the do fn, do we need to initialize them in the setup
code? My concern is that if we have this initialization in each do fn, we might
overwhelm the memory. I think we can pass them as the parameter? Do you have a
recommissioned way to do that not to hurt the performance?
Will try that out to see if this is workable!
Siyu
>
> On May 6, 2022, at 3:35 PM, Pablo Estrada <pabl...@google.com> wrote:
>
>
> ah - no, I am not aware of anyone running side containers.
>
> You can add libraries to your container that will be available, so that you
> can import prometheus client libraries in your DoFns and export metrics
> directly 'by hand' (e.g. processElement(..) {
> prometheusCliient.reportMetric("name", TYPE, value); })
>
> If you need to run another program in your worker - I suppose you can add new
> binaries that run next to your Beam worker processes (in the same container,
> and you have to start them up in the entrypoint of your Dockerfile). I've
> never seen anyone do this, but it sounds quite doable.
>
> Do any of these options work for you?
> Best
> -P.
>
>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <siyu...@unity3d.com> wrote:
>> Hi Pablo,
>>
>> No worries at all!
>>
>> Was wondering if “add any dependencies” means to add a side container to the
>> dataflow runner? Like prometheus scraper?
>>
>> If so, is it only working for dataflow runner v2? We have not upgraded to v2
>> yet so it might be hard to do that in v1.
>>
>> Thanks so much!
>>
>> Siyu
>>
>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pabl...@google.com> wrote:
>>>
>>> Sorry about the delay!
>>>
>>> Yes, you can add any dependencies to your image - and you can add custom
>>> reporting of metrics that you're tracking directly. That may help?
>>>
>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>> Hi Jeff,
>>>>
>>>> Thanks so much for your quick responses. It is unfortunate that histogram
>>>> is unavailable in dataflow. Do you know if there are any workaround? Or do
>>>> you think it is plausible if we can use runner v2 and customize the image
>>>> with Prometheus exporter?
>>>>
>>>> Thanks again!
>>>> Siyu
>>>>
>>>>>
>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jklu...@mozilla.com> wrote:
>>>>>>
>>>>>
>>>>> Siyu - The Beam metrics interface includes the Distribution metric type
>>>>> which can be used for histograms:
>>>>>
>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>
>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the
>>>>> reported values are MAX, MIN, MEAN, and COUNT, so no support for
>>>>> finer-grained percentiles:
>>>>>
>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>
>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>>> Hi Beam community,
>>>>>>
>>>>>> I am wondering if there is histogram metrics available (or alternative
>>>>>> recommendations) for showing up quantiles. We have counter metrics
>>>>>> already but we would also like to see some quantiles for different
>>>>>> values.
>>>>>>
>>>>>> Thanks a lot!
>>>>>> Siyu
>>
