Re: Simple stateful polling source

Small correction to what I said: Sources have to implement ParallelSourceFunction in order to be run with a higher parallelism. The javadocs for the RichSourceFunction are /somewhat /incorrect, but in a sense also correct. This is because you can have a RichSourceFunction that also implements

Re: Simple stateful polling source

Hi Chesnay, > On Jun 19, 2019, at 6:05 AM, Chesnay Schepler wrote: > > A (Rich)SourceFunction that does not implement RichParallelSourceFunction is > always run with a parallelism of 1. RichSourceFunction

Re: Simple stateful polling source

Ok great! Thanks everybody for the support On Wed, Jun 19, 2019 at 3:05 PM Chesnay Schepler wrote: > A (Rich)SourceFunction that does not implement RichParallelSourceFunction > is always run with a parallelism of 1. > > On 19/06/2019 14:36, Flavio Pompermaier wrote: > > My sourcefunction is intr

Re: Simple stateful polling source

A (Rich)SourceFunction that does not implement RichParallelSourceFunction is always run with a parallelism of 1. On 19/06/2019 14:36, Flavio Pompermaier wrote: My sourcefunction is intrinsically single-thread. Is there a way to force this aspect? I can't find a real difference between a RichPar

Re: Simple stateful polling source

My sourcefunction is intrinsically single-thread. Is there a way to force this aspect? I can't find a real difference between a RichParallelSourceFunction and a RichSourceFunction. Is this last (RichSourceFunction) implicitly using parallelism = 1? On Wed, Jun 19, 2019 at 2:25 PM Chesnay Schepler

Re: Simple stateful polling source

It returns a list of states so that state can be re-distributed if the parallelism changes. If you hard-code the interface to return a single value then you're implicitly locking the parallelism. When you reduce the parallelism you'd no longer be able to restore all state, since you have less

Re: Simple stateful polling source

It's not clear to me why the source checkpoint returns a list of object...when it could be useful to use a list instead of a single value? The documentation says The returned list should contain one entry for redistributable unit of state" but this is not very clear to me.. Best, Flavio On Wed, J

Re: Simple stateful polling source

This looks fine to me. What exactly were you worried about? On 19/06/2019 12:33, Flavio Pompermaier wrote: Hi to all, in my use case I have to ingest data from a rest service, where I periodically poll the data (of course a queue would be a better choice but this doesn't depend on me). So I