I think case in point dependency that would benefit from this testing is grpcio, which includes pre-releases, and broke us and multiple of it's released versions were yanked. https://pypi.org/project/grpcio/#history .
We can look at how grpcio affected Beam previously. Couple of issues: - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests - https://github.com/apache/beam/issues/23734 -- affected MacOS suites - https://github.com/apache/beam/issues/22159 -- (not detected by us, but potentially could have affected a performance test). I'm afraid a dedicated suite may not give us desired test coverage to catch regression at RC stage. On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <dev@beam.apache.org> wrote: > Thanks Anand, > > This would be very helpful to avoid experiencing multiple time ( > https://s.apache.org/beam-python-dependencies-pm). One thing to note is > that Beam Jenkins CI is experiencing many issues recently, mostly due to > that multiple Jenkins plugins does not scale (draining GitHub API call > limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins > if going ahead with Option 1. As we have started GitHub Action migration, > is it considered to add these new tests to GitHub Action? > > Best, > Yi > > On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having >> the clear signal of the normal suite succeeding and the prerelease one >> failing would be helpful and there shouldn't be too much additional code >> necessary. That makes it really easy to treat the prerelease suite as a (at >> least temporary) signal on needing upper bounds on our dependencies. >> >> Thanks, >> Danny >> >> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev < >> dev@beam.apache.org> wrote: >> >>> Hi all, >>> >>> For Apache Beam Python we are considering using pre-released >>> dependencies for unit testing by using the --pre flag to install >>> pre-released dependencies of packages. >>> >>> We believe that using pre-released dependencies may help us to identify >>> and resolve bugs more quickly, and to take advantage of new features or bug >>> fixes that are not yet available in stable releases. However, we also >>> understand that using pre-released dependencies may introduce new risks and >>> challenges, including potential code duplication and stability issues. >>> >>> Before proceeding, we wanted to get your feedback on this approach. >>> >>> 1. Create a new PreCommit test suite and a PostCommit test suite that >>> runs tests by installing pre-released dependencies. >>> >>> Pros: >>> >>> - stable and pre-released test suites are separate and it will be >>> easier to debug if the pre-released test suite fails. >>> >>> Cons: >>> >>> - More test infra code to maintain. More tests to monitor. >>> >>> >>> 2. Make use of the current PreCommit and PostCommit test suite and >>> modify it so that it installs pre-released dependencies. >>> >>> Pros: >>> >>> - Less infra code and less tests to monitor. >>> >>> Cons: >>> >>> - Leads to noisy test signals if the pre-release candidate is >>> unstable. >>> >>> I am in favor of approach 1 since this approach would ensure that any >>> issues encountered during pre-release testing do not impact the stable >>> release environment, and vice versa. >>> >>> If you have experience or done any testing work using pre-released >>> dependencies, please let me know if you took any different approaches. It >>> will be really helpful. >>> >>> Thanks, >>> Anand >>> >>
