Hello Everyone I have a question, I have on my pipeline one side input that fetches some configurations from an API endpoint each 30 seconds, my question is this.
I have something similar to what is showed in the side input patterns documentation PCollectionView<Map<String, String>> map = p.apply(GenerateSequence.from(0).withRate(1, Duration.standardSeconds(5L))) .apply( ParDo.of( new DoFn<Long, Map<String, String>>() { @ProcessElement public void process( @Element Long input, @Timestamp Instant timestamp, OutputReceiver<Map<String, String>> o) { call HTTP endpoint here!! } })) .apply( Window.<Map<String, String>>into(new GlobalWindows()) .triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane())) .discardingFiredPanes()) .apply(Latest.globally()) .apply(View.asSingleton()); What happens if for example the HTTP endpoint takes time to respond due some network issues and/or the amount of data. Is this gonna introduce delays on my main pipeline? Is the main pipeline blocked until the pardo that processes the side input ends? I don't care too much about the consistency here, I mean if the configuration changed in the Time T1 I don't care if some registries with T2 timestamp are processed with the configuration version of T1. Regards.