Hey all! After some more rigorous testing on real environments we have found some critical issues with the runtime config observe feature (https://issues.apache.org/jira/browse/FLINK-35746) that causes some major problems in combination with application health checks. Ultimately it causes deployment errors, redeploys healthy applications with configs that fail subsequently in many cases, inconsistencies during the observation etc.
Turns out that we have overlooked some complexity in the implementation and the more we look at it the more issues we find. Given these problems I think the best course of action is to revert this feature on both main and release-1.16. I would suggest proceeding immediately with a 1.16.1 patch release to resolve this issue. I am going to prepare the RC ASAP, only including the revert and 1-2 select more critical fixes that landed on main in the meantime. Cheers Gyula
