Thanks for doing this Anand, I'm +1 on option 1 as well - I think having the clear signal of the normal suite succeeding and the prerelease one failing would be helpful and there shouldn't be too much additional code necessary. That makes it really easy to treat the prerelease suite as a (at least temporary) signal on needing upper bounds on our dependencies.
Thanks, Danny On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <dev@beam.apache.org> wrote: > Hi all, > > For Apache Beam Python we are considering using pre-released dependencies > for unit testing by using the --pre flag to install pre-released > dependencies of packages. > > We believe that using pre-released dependencies may help us to identify > and resolve bugs more quickly, and to take advantage of new features or bug > fixes that are not yet available in stable releases. However, we also > understand that using pre-released dependencies may introduce new risks and > challenges, including potential code duplication and stability issues. > > Before proceeding, we wanted to get your feedback on this approach. > > 1. Create a new PreCommit test suite and a PostCommit test suite that runs > tests by installing pre-released dependencies. > > Pros: > > - stable and pre-released test suites are separate and it will be > easier to debug if the pre-released test suite fails. > > Cons: > > - More test infra code to maintain. More tests to monitor. > > > 2. Make use of the current PreCommit and PostCommit test suite and modify > it so that it installs pre-released dependencies. > > Pros: > > - Less infra code and less tests to monitor. > > Cons: > > - Leads to noisy test signals if the pre-release candidate is > unstable. > > I am in favor of approach 1 since this approach would ensure that any > issues encountered during pre-release testing do not impact the stable > release environment, and vice versa. > > If you have experience or done any testing work using pre-released > dependencies, please let me know if you took any different approaches. It > will be really helpful. > > Thanks, > Anand >
