Hi Johannes,
L20 in your code will make it run forever so it is better to put a finite
number there, like
`GenerateSequence.from(0).to(1).withRate(1, Duration.standardSeconds(2))`
-siyu
> On Sep 10, 2021, at 6:06 AM, Johannes Frey wrote:
>
> Hi everybody,
>
> I'm currently having a hard tim
Hi Luke,
I had the same issue before. Was wondering if it is possible to use a
different timestamp as the watermark. Thanks!
-siyu
On Fri, Nov 19, 2021 at 11:42 AM Luke Cwik wrote:
> The issue seems to be that when doing a backfill Kafka is using wall time
> to estimate the watermark and then
Hi all,
Was wondering if we can set up a system property to print out detailed errors
when tests throwing exceptions / errors. I remember that I set it up before but
forget the exact name.
Appreciate any help!
-siyu
Hi Reza,
Just read a blog post[1] by you two years ago and you mentioned
> Because this pattern uses a global-window SideInput, matching to elements
> being processed will be nondeterministic.
Do you mean two workers working on the same windowed main input and use
different global windowed si
Hi Beam community,
I am wondering if there is histogram metrics available (or alternative
recommendations) for showing up quantiles. We have counter metrics already but
we would also like to see some quantiles for different values.
Thanks a lot!
Siyu
epend on the runner. For Cloud Dataflow, the reported
> values are MAX, MIN, MEAN, and COUNT, so no support for finer-grained
> percentiles:
>
> https://cloud.google.com/dataflow/docs/guides/using-cloud-monitoring#custom_metrics
>
>> On Mon, Apr 4, 2022 at 7:45 PM Siyu Lin wr
Hi Beam community,
We have a batch pipeline which does not run regularly. Recently we
have upgraded to Beam 2.36 and this broke the FileIO WriteDynamic
process.
We are using Dataflow Runner, and the errors are like this when there
are multiple workers:
Error message from worker: java.lang.NoClas
Hi Reuven,
Do you mean we should have coder explicitly defined for all input types and
output types in chaining do fns? Do we also need to have comparedTo and equals
defined as well?
thanks again!
Siyu
> On May 6, 2022, at 12:23 PM, Reuven Lax wrote:
>
>
> Could be - I would check the impl
> On May 6, 2022, at 11:57 AM, Pablo Estrada wrote:
>
> Sorry about the delay!
>
> Yes, you can add any dependencies to your image - and you can add custom
> reporting of metrics that you're tracking directly. That may help?
>
> On Mon, Apr 4, 2022 at 7
worker - I suppose you can add new
> binaries that run next to your Beam worker processes (in the same container,
> and you have to start them up in the entrypoint of your Dockerfile). I've
> never seen anyone do this, but it sounds quite doable.
>
> Do any of these opti
sorry for typo “recommissioned way” should be “recommended way”
> On May 6, 2022, at 4:31 PM, Siyu Lin wrote:
>
>
> Hi Pablo,
>
> Thanks so much for your explanation!
>
> Also, for prom client in the do fn, do we need to initialize them in the
> setup code? My
by many threads at the same time - so you need to make sure that
> PrometheusClient is thread safe, or you need to synchronize accesses to it.
> (maybe it supports some kind of async call?)
>
> Let me know if that makes sense?
> Best
> -P.
>
>> On Fri, May 6, 2022 at
12 matches
Mail list logo
