Got it! Thanks so much, and will try it out!
-siyu
>
> On May 6, 2022, at 4:53 PM, Pablo Estrada <pabl...@google.com> wrote:
>
>
> Hi Siyu!
> Yeah, that's a good question!
>
> Depending on the properties of the client that you're using, you can create
> it as a class/static variable rather than an instance variable:
>
> class MyPrometheusDoFn extends DoFn<...> {
> static PrometheusClient client = Prometheus.client(); // Or whatever
> }
>
> You just need to consider that in this scenario, the variable `client` may be
> used by many threads at the same time - so you need to make sure that
> PrometheusClient is thread safe, or you need to synchronize accesses to it.
> (maybe it supports some kind of async call?)
>
> Let me know if that makes sense?
> Best
> -P.
>
>> On Fri, May 6, 2022 at 4:33 PM Siyu Lin <siyu...@unity3d.com> wrote:
>> sorry for typo “recommissioned way” should be “recommended way”
>>
>>
>>>> On May 6, 2022, at 4:31 PM, Siyu Lin <siyu...@unity3d.com> wrote:
>>>>
>>>
>>> Hi Pablo,
>>>
>>> Thanks so much for your explanation!
>>>
>>> Also, for prom client in the do fn, do we need to initialize them in the
>>> setup code? My concern is that if we have this initialization in each do
>>> fn, we might overwhelm the memory. I think we can pass them as the
>>> parameter? Do you have a recommissioned way to do that not to hurt the
>>> performance?
>>>
>>> Will try that out to see if this is workable!
>>>
>>> Siyu
>>>
>>>>
>>>>> On May 6, 2022, at 3:35 PM, Pablo Estrada <pabl...@google.com> wrote:
>>>>>
>>>>
>>>> ah - no, I am not aware of anyone running side containers.
>>>>
>>>> You can add libraries to your container that will be available, so that
>>>> you can import prometheus client libraries in your DoFns and export
>>>> metrics directly 'by hand' (e.g. processElement(..) {
>>>> prometheusCliient.reportMetric("name", TYPE, value); })
>>>>
>>>> If you need to run another program in your worker - I suppose you can add
>>>> new binaries that run next to your Beam worker processes (in the same
>>>> container, and you have to start them up in the entrypoint of your
>>>> Dockerfile). I've never seen anyone do this, but it sounds quite doable.
>>>>
>>>> Do any of these options work for you?
>>>> Best
>>>> -P.
>>>>
>>>>> On Fri, May 6, 2022 at 3:16 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>> Hi Pablo,
>>>>>
>>>>> No worries at all!
>>>>>
>>>>> Was wondering if “add any dependencies” means to add a side container to
>>>>> the dataflow runner? Like prometheus scraper?
>>>>>
>>>>> If so, is it only working for dataflow runner v2? We have not upgraded to
>>>>> v2 yet so it might be hard to do that in v1.
>>>>>
>>>>> Thanks so much!
>>>>>
>>>>> Siyu
>>>>>
>>>>>> On May 6, 2022, at 11:57 AM, Pablo Estrada <pabl...@google.com> wrote:
>>>>>>
>>>>>> Sorry about the delay!
>>>>>>
>>>>>> Yes, you can add any dependencies to your image - and you can add custom
>>>>>> reporting of metrics that you're tracking directly. That may help?
>>>>>>
>>>>>>> On Mon, Apr 4, 2022 at 7:10 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> Thanks so much for your quick responses. It is unfortunate that
>>>>>>> histogram is unavailable in dataflow. Do you know if there are any
>>>>>>> workaround? Or do you think it is plausible if we can use runner v2 and
>>>>>>> customize the image with Prometheus exporter?
>>>>>>>
>>>>>>> Thanks again!
>>>>>>> Siyu
>>>>>>>
>>>>>>>>
>>>>>>>>> On Apr 4, 2022, at 5:00 PM, Jeff Klukas <jklu...@mozilla.com> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Siyu - The Beam metrics interface includes the Distribution metric
>>>>>>>> type which can be used for histograms:
>>>>>>>>
>>>>>>>> https://beam.apache.org/documentation/programming-guide/#types-of-metrics
>>>>>>>>
>>>>>>>> Particulars of support depend on the runner. For Cloud Dataflow, the
>>>>>>>> reported values are MAX, MIN, MEAN, and COUNT, so no support for
>>>>>>>> finer-grained percentiles:
>>>>>>>>
>>>>>>>> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>>>>>>>>
>>>>>>>>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin <siyu...@unity3d.com> wrote:
>>>>>>>>> Hi Beam community,
>>>>>>>>>
>>>>>>>>> I am wondering if there is histogram metrics available (or
>>>>>>>>> alternative recommendations) for showing up quantiles. We have
>>>>>>>>> counter metrics already but we would also like to see some quantiles
>>>>>>>>> for different values.
>>>>>>>>>
>>>>>>>>> Thanks a lot!
>>>>>>>>> Siyu
>>>>>
