Sounds good, thanks! Best, Yi
On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <ananding...@google.com> wrote: > @Yi Hu <ya...@google.com> I think adding them to Jenkins or github > actions is okay with me. With Github actions, since we don't use self > hosted runners yet, I worry that action workers might get queued up. > > Also, I plan to not run these on every commit but run it as a cron > job(maybe once per day) and also as trigger phrases and only on the lowest > and highest python version. Also, migrating this workflow to jenkins would > be trivial in the future once beam starts the migration. For now, I think > it might be best to run on jenkins. > > On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> I think case in point dependency that would benefit from this testing is >> grpcio, which includes pre-releases, and broke us and multiple of it's >> released versions were yanked. https://pypi.org/project/grpcio/#history . >> >> We can look at how grpcio affected Beam previously. Couple of issues: >> >> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests >> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites >> - https://github.com/apache/beam/issues/22159 -- (not detected by us, >> but potentially could have affected a performance test). >> >> I'm afraid a dedicated suite may not give us desired test coverage to >> catch regression at RC stage. >> >> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <dev@beam.apache.org> >> wrote: >> >>> Thanks Anand, >>> >>> This would be very helpful to avoid experiencing multiple time ( >>> https://s.apache.org/beam-python-dependencies-pm). One thing to note is >>> that Beam Jenkins CI is experiencing many issues recently, mostly due to >>> that multiple Jenkins plugins does not scale (draining GitHub API call >>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins >>> if going ahead with Option 1. As we have started GitHub Action migration, >>> is it considered to add these new tests to GitHub Action? >>> >>> Best, >>> Yi >>> >>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think >>>> having the clear signal of the normal suite succeeding and the prerelease >>>> one failing would be helpful and there shouldn't be too much additional >>>> code necessary. That makes it really easy to treat the prerelease suite as >>>> a (at least temporary) signal on needing upper bounds on our dependencies. >>>> >>>> Thanks, >>>> Danny >>>> >>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> Hi all, >>>>> >>>>> For Apache Beam Python we are considering using pre-released >>>>> dependencies for unit testing by using the --pre flag to install >>>>> pre-released dependencies of packages. >>>>> >>>>> We believe that using pre-released dependencies may help us to >>>>> identify and resolve bugs more quickly, and to take advantage of new >>>>> features or bug fixes that are not yet available in stable releases. >>>>> However, we also understand that using pre-released dependencies may >>>>> introduce new risks and challenges, including potential code duplication >>>>> and stability issues. >>>>> >>>>> Before proceeding, we wanted to get your feedback on this approach. >>>>> >>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that >>>>> runs tests by installing pre-released dependencies. >>>>> >>>>> Pros: >>>>> >>>>> - stable and pre-released test suites are separate and it will be >>>>> easier to debug if the pre-released test suite fails. >>>>> >>>>> Cons: >>>>> >>>>> - More test infra code to maintain. More tests to monitor. >>>>> >>>>> >>>>> 2. Make use of the current PreCommit and PostCommit test suite and >>>>> modify it so that it installs pre-released dependencies. >>>>> >>>>> Pros: >>>>> >>>>> - Less infra code and less tests to monitor. >>>>> >>>>> Cons: >>>>> >>>>> - Leads to noisy test signals if the pre-release candidate is >>>>> unstable. >>>>> >>>>> I am in favor of approach 1 since this approach would ensure that any >>>>> issues encountered during pre-release testing do not impact the stable >>>>> release environment, and vice versa. >>>>> >>>>> If you have experience or done any testing work using pre-released >>>>> dependencies, please let me know if you took any different approaches. It >>>>> will be really helpful. >>>>> >>>>> Thanks, >>>>> Anand >>>>> >>>>
