Reading the below two statements I conclude that
CheckpointMark.finalizeCheckpoint() will be called in order, unless there is a
failure.
What happens in a failure?
What happens to subsequent checkpoints in the case of a checkpoint failure?
How do I prevent event re-ordering in the case of a check
Does anyone actually use Streaming Autoscaling with cloud Dataflow? I have
seen scale-ups based on CPU but never on backlog. Now I do not see scale up
events at all. If this works can you please point me to a working example.
On 2019/01/09 20:09:46, Ken Barr wrote:
> Hello
>
> I
an indecation
on what was going on. My backlog method was returning 0 which was holding down
auto-scaling. Basically everything was working as designed, just difficult to
troubleshoot.
On 2019/01/09 20:09:46, Ken Barr wrote:
> Hello
>
> I have been directed to this mail list by p
I have a custom UnboundedSource IO that reads from a series of messaging
queues. I have implemented this such that the IO takes a list of queues and
expands a UnboundedSource/UnboundedReader for each queue.
If I use autoscaling with maxNumWorkers <= # number of queues everything works
well.
; b) if you want to limit parallelism, that should be possible in the
> split() method - you can return collection containing only (this) if
> there is no more readers
>
> Hope this helps, please feel free to ask more details if needed.
>
> Best,
>
> Jan
>
>
if
> there is no more readers
>
> Hope this helps, please feel free to ask more details if needed.
>
> Best,
>
> Jan
>
> On 9/19/19 4:47 PM, Ken Barr wrote:
> > I have a custom UnboundedSource IO that reads from a series of messaging
> > queues. I have impl
works and that I can see the runtime exception.
When I run in the situation where deadlock occurs I never see the runtime
exception, ie. The Reader that is holding exclusivity never has it's advance()
method called after scaleup.
On 2019/09/19 17:51:35, Ken Barr wrote:
> The Start() see
I might have to resort to this. I am reading from queue in a timed manor and
properly returning false when read timeout. I have implemented a set of
statistic to enumerate advance() behavior, number of times it successfully
reads a message, fails to read a message etc.
On 2019/09/19 19:45:49,
return false
> otherwise)
>
> Each blocking call in advance() is potential source of distributed
> deadlocks which might be what you observe.
>
> Jan
>
> On 9/24/19 4:07 PM, Ken Barr wrote:
> > I might have to resort to this. I am reading from queue in a timed
I am currently developing an IO that I would like to eventually submit to
Apache Beam project. The IO itself is Apache2.0 licensed.
Does every chained dependency I use need to be opensource?
If yes, how is this usually proven?
Is it enough that only Maven dependencies are used?
Hello
I have been directed to this mail list by posting to Stack Overflow. If this
is inappropriate use of mail list please let me
know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
I am creating a custom IO connector in Java that runs
11 matches
Mail list logo
