Hi Alex and Yang, Apologies for the delayed reply…
Yes I’d also love to share/exchange ideas. I’ll have a FLIP ready between this week and the next and we could continue the conversation there, does that sound good? Alex N. Indeed your solution sounds extremely similar! Let’s see what common ground we’re both covering. Thank you, Sergio > On Dec 19, 2024, at 7:33 AM, Alex Nitavsky <alexnitav...@gmail.com> wrote: > > Thanks Alexander for looking into email. > > - Should this reusable ProcessFunction filter out data or simply tag >> them? For example, filtering data for green deployments would be >> suitable >> in the case of Async IO outputs. > > > Consider a Flink job that mutates data in an external service using Async > IO. If we are running this job with a blue-green deployment, it might be > crucial to ensure that the Green job does not produce such mutations. This > could be due to performance or quality considerations. Then there are could > be different mechanism/APIs to provide such behaviour, e.g. wrapping the > data into the tuple with color and it is up to user to have specific logic > in order to process blue/green messages. > > And there are another question, which have been lost due to the formatting > issues: > > - Have you considered adding a validation mechanism for Blue-Green > deployments? For instance, you could allow users to perform automatic > validation before promoting the Green Flink job to Blue. Simple solution: > After deploying of the Green job, k8s operator could wait for an external > ConfigMap change or a property change in the FlinkBlueGreenDeployment CRD > to initiate the promotion. Such ConfigMap change/CRD change, could be > initiated by another external system which ensures that Green job is > performing correctly. > > Regards > Alex > > > On Tue, Dec 17, 2024 at 2:16 PM Alexander Fedulov < > alexander.fedu...@gmail.com> wrote: > >> Hi Alex, >> >> - Should this reusable ProcessFunction filter out data or simply tag >>> them? For example, filtering data for green deployments would be >>> suitable >>> in the case of Async IO outputs. >> >> >> Could you clarify the idea a bit more, it is not quite clear what you mean. >> >> - Should we also modify the Source API to allow it to provide COLOR to >>> the Source functions? This could be used, for instance, to generate a >>> different groupId for Kafka. >> >> >> That is a good point, and can definitely be useful, especially given the >> concerns that both jobs can consume at significantly different rates, >> particularly if the blue job is backpressured by the sink. >> >> Best, >> Alex >> >> On Mon, 9 Dec 2024 at 14:36, Yang LI <yang.hunter...@gmail.com> wrote: >> >>> Hello Sergio and Jose, >>> >>> Apologies for jumping into the conversation. I’m very interested in what >>> you’ve learned about blue/green Flink deployments. I’m currently working >>> on >>> a project aimed at achieving no-downtime Flink deployments, so your >>> insights would be highly valuable. >>> >>> Thank you, >>> Yang >>> >>
