sorry for typo “recommissioned way” should be “recommended way”
> On May 6, 2022, at 4:31 PM, Siyu Lin <siyu...@unity3d.com> wrote:
>
>
> Hi Pablo,
>
> Thanks so much for your explanation!
>
> Also, for prom client in the do fn, do we need to initialize them in the
> setup code? My concern is that if we have this initialization in each do fn,
> we might overwhelm the memory. I think we can pass them as the parameter? Do
> you have a recommissioned way to do that not to hurt the performance?
>
> Will try that out to see if this is workable!
>
> Siyu
>
>>
>>> On May 6, 2022, at 3:35 PM, Pablo Estrada <pabl...@google.com> wrote:
>>>
>>
>> ah - no, I am not aware of anyone running side containers.
>>
>> You can add libraries to your container that will be available, so that you
>> can import prometheus client libraries in your DoFns and export metrics
>> directly 'by hand' (e.g. processElement(..) {
>> prometheusCliient.reportMetric("name", TYPE, value); })
>>
>> If you need to run another program in your worker - I suppose you can add
>> new binaries that run next to your Beam worker processes (in the same
>> container, and you have to start them up in the entrypoint of your
>> Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>>
>> Do any of these options work for you?
>> Best
>> -P.
>>
>>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>> Hi Pablo,
>>>
>>> No worries at all!
>>>
>>> Was wondering if “add any dependencies” means to add a side container to
>>> the dataflow runner? Like prometheus scraper?
>>>
>>> If so, is it only working for dataflow runner v2? We have not upgraded to
>>> v2 yet so it might be hard to do that in v1.
>>>
>>> Thanks so much!
>>>
>>> Siyu
>>>
>>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pabl...@google.com> wrote:
>>>>
>>>> Sorry about the delay!
>>>>
>>>> Yes, you can add any dependencies to your image - and you can add custom
>>>> reporting of metrics that you're tracking directly. That may help?
>>>>
>>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>> Hi Jeff,
>>>>>
>>>>> Thanks so much for your quick responses. It is unfortunate that histogram
>>>>> is unavailable in dataflow. Do you know if there are any workaround? Or
>>>>> do you think it is plausible if we can use runner v2 and customize the
>>>>> image with Prometheus exporter?
>>>>>
>>>>> Thanks again!
>>>>> Siyu
>>>>>
>>>>>>
>>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jklu...@mozilla.com> wrote:
>>>>>>>
>>>>>>
>>>>>> Siyu - The Beam metrics interface includes the Distribution metric type
>>>>>> which can be used for histograms:
>>>>>>
>>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>>
>>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the
>>>>>> reported values are MAX, MIN, MEAN, and COUNT, so no support for
>>>>>> finer-grained percentiles:
>>>>>>
>>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>>
>>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>>>> Hi Beam community,
>>>>>>>
>>>>>>> I am wondering if there is histogram metrics available (or alternative
>>>>>>> recommendations) for showing up quantiles. We have counter metrics
>>>>>>> already but we would also like to see some quantiles for different
>>>>>>> values.
>>>>>>>
>>>>>>> Thanks a lot!
>>>>>>> Siyu
>>>
