For posterity here met with Nick over GVC, the issue is coming from the file Bruno identified above, and is a warning for their custom histogram logic.
Svetak Sundhar Technical Solutions Engineer, Data s <nellywil...@google.com>vetaksund...@google.com On Fri, Mar 3, 2023 at 1:40 PM Bruno Volpato via user <user@beam.apache.org> wrote: > Found some more info, sorry for the chopping. > > Are you using *bigqueryio* or *bigquery_tools* somehow? > If so, biguquery_tools defines a histogram using 20 buckets of 3 seconds > each to export latencies (see > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L347-L351 > ). > > Besides having requests with latency >60s not properly recorded (and > warning logs), this shouldn't have any impact. > > > On Fri, Mar 3, 2023 at 1:31 PM Bruno Volpato <bvolp...@google.com> wrote: > >> Hi Nick, >> >> This seems to come from utils/histogram.py >> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/histogram.py#L75-L89>. >> Any chance that you are initializing it in a way that defines bounds up to >> 60,000 but invoking record() with out of bounds value? >> >> Best, >> Bruno >> >> On Fri, Mar 3, 2023 at 1:26 PM Nick Edwards <nwedweards....@gmail.com> >> wrote: >> >>> Hello! I’m investigating a warning we’re receiving through Google >>> Cloud’s Monitoring. The warning reads "record is out of upper bound 60000: >>> " and then includes a 5 or 6 digit integer. I’m having trouble finding any >>> info on this sort of behavior. Any insight or assistance you can offer is >>> greatly appreciated! >>> >>> >>> [image: image.png] >>> >>> >>> >>> >>
