Friendly ping:) If anyone has experience testing the pre-released versions, your feedback would be much appreciated.
Thanks, Anand On Wed, Apr 12, 2023 at 5:07 PM Yi Hu <ya...@google.com> wrote: > Sounds good, thanks! > > Best, > Yi > > On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <ananding...@google.com> > wrote: > >> @Yi Hu <ya...@google.com> I think adding them to Jenkins or github >> actions is okay with me. With Github actions, since we don't use self >> hosted runners yet, I worry that action workers might get queued up. >> >> Also, I plan to not run these on every commit but run it as a cron >> job(maybe once per day) and also as trigger phrases and only on the lowest >> and highest python version. Also, migrating this workflow to jenkins would >> be trivial in the future once beam starts the migration. For now, I think >> it might be best to run on jenkins. >> >> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> I think case in point dependency that would benefit from this testing is >>> grpcio, which includes pre-releases, and broke us and multiple of it's >>> released versions were yanked. https://pypi.org/project/grpcio/#history >>> . >>> >>> We can look at how grpcio affected Beam previously. Couple of issues: >>> >>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests >>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites >>> - https://github.com/apache/beam/issues/22159 -- (not detected by us, >>> but potentially could have affected a performance test). >>> >>> I'm afraid a dedicated suite may not give us desired test coverage to >>> catch regression at RC stage. >>> >>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <dev@beam.apache.org> >>> wrote: >>> >>>> Thanks Anand, >>>> >>>> This would be very helpful to avoid experiencing multiple time ( >>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note >>>> is that Beam Jenkins CI is experiencing many issues recently, mostly due to >>>> that multiple Jenkins plugins does not scale (draining GitHub API call >>>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins >>>> if going ahead with Option 1. As we have started GitHub Action migration, >>>> is it considered to add these new tests to GitHub Action? >>>> >>>> Best, >>>> Yi >>>> >>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think >>>>> having the clear signal of the normal suite succeeding and the prerelease >>>>> one failing would be helpful and there shouldn't be too much additional >>>>> code necessary. That makes it really easy to treat the prerelease suite as >>>>> a (at least temporary) signal on needing upper bounds on our dependencies. >>>>> >>>>> Thanks, >>>>> Danny >>>>> >>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> For Apache Beam Python we are considering using pre-released >>>>>> dependencies for unit testing by using the --pre flag to install >>>>>> pre-released dependencies of packages. >>>>>> >>>>>> We believe that using pre-released dependencies may help us to >>>>>> identify and resolve bugs more quickly, and to take advantage of new >>>>>> features or bug fixes that are not yet available in stable releases. >>>>>> However, we also understand that using pre-released dependencies may >>>>>> introduce new risks and challenges, including potential code duplication >>>>>> and stability issues. >>>>>> >>>>>> Before proceeding, we wanted to get your feedback on this approach. >>>>>> >>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that >>>>>> runs tests by installing pre-released dependencies. >>>>>> >>>>>> Pros: >>>>>> >>>>>> - stable and pre-released test suites are separate and it will be >>>>>> easier to debug if the pre-released test suite fails. >>>>>> >>>>>> Cons: >>>>>> >>>>>> - More test infra code to maintain. More tests to monitor. >>>>>> >>>>>> >>>>>> 2. Make use of the current PreCommit and PostCommit test suite and >>>>>> modify it so that it installs pre-released dependencies. >>>>>> >>>>>> Pros: >>>>>> >>>>>> - Less infra code and less tests to monitor. >>>>>> >>>>>> Cons: >>>>>> >>>>>> - Leads to noisy test signals if the pre-release candidate is >>>>>> unstable. >>>>>> >>>>>> I am in favor of approach 1 since this approach would ensure that any >>>>>> issues encountered during pre-release testing do not impact the stable >>>>>> release environment, and vice versa. >>>>>> >>>>>> If you have experience or done any testing work using pre-released >>>>>> dependencies, please let me know if you took any different approaches. It >>>>>> will be really helpful. >>>>>> >>>>>> Thanks, >>>>>> Anand >>>>>> >>>>>
