I also make PRs to fix CI for branch-4.15 and branch-4.16 since we will be maintaining those branches for some time. https://github.com/apache/bookkeeper/pull/4207 for branch-4.15 https://github.com/apache/bookkeeper/pull/4206 for branch-4.16 These PR backport changes from master branch to the maintenance branches to make the CI run all tests, including integration tests. Please review!
-Lari On 2024/02/08 10:59:17 Lari Hotari wrote: > Finally all passing and merged! Thanks for the reviews. > > The docs for running the integration tests with Mac M1/M2/M3 is at > https://github.com/apache/bookkeeper/tree/master/tests#workaround-for-running-the-tests-with-mac-apple-silicon > . The plan is to replace Arquillian Cube with Testcontainers and Junit 5 for > the integration tests. Until that is in place, a workaround is needed for Mac > OS on Apple Silicon. > > -Lari > > On 2024/02/08 07:56:47 Lari Hotari wrote: > > After 91 commits, I'm very close in getting integration tests running and > > fixed in CI. > > > > Integration tests have been ignored in apache/bookkeeper CI for a very long > > time. Here's some evidence in the comment I added to #3675: > > https://github.com/apache/bookkeeper/pull/3675#issuecomment-1933485215 > > To get integration tests to pass, I had to fix a bug in "bkctl ledger > > simpletest" and it appears that it was already merged as invalid in > > November 2022. 😱 > > > > The good news is that integration tests are now passing in the PR builds > > for https://github.com/apache/bookkeeper/pull/4197. 😮💨 > > > > -Lari > > > > On 2024/02/06 23:10:35 Lari Hotari wrote: > > > After some night hacking (58 commits), I'm finally making progress in > > > fixing Bookkeeper CI. It's a long story to explain all details, so it's > > > better to just check the PR: > > > https://github.com/apache/bookkeeper/pull/4197 > > > > > > Btw. I also found the reason why tests didn't run for the integration > > > tests at all. Maven surefire had picked TestNG provider for some reason > > > and didn't find any tests. Perhaps some dependency pulled in TestNG and > > > that caused the trouble. > > > > > > Besides fixing a lot of issues across a lot of different areas related to > > > the integration & backwards compatibility tests, I also documented how > > > the old Arquillian Cube Docker tests > > > can be run on Mac Apple Silicon (M1/M2/M3). > > > https://github.com/lhotari/bookkeeper/blob/3e13e0247d1c570c7cae67dadf200d6f3b095d7a/tests/README.md > > > This helped resolve the remaining issues since I could simple debug > > > things in my local development environment on Mac M3. > > > > > > The plan is to replace Arquillian Cube with Testcontainers & Junit 5 > > > after this PR has been completed with reviews and merged. > > > Any other volunteers that would be interested in participating in this > > > work? > > > > > > -Lari > > > > > > On 2024/02/02 14:21:30 Lari Hotari wrote: > > > > It has been a rabbit hole to fix the CI issues. > > > > I have put a lot of commits into > > > > https://github.com/apache/bookkeeper/pull/4197/commits to handle many > > > > problems. The integration tests run now, but there are failures. I > > > > didn't have time to continue investigating and fixing problems. I'll > > > > continue later. > > > > > > > > I noticed that the circe-checksum native library has been broken too. > > > > Fixing it required adding this to nar-maven-plugin's configuration's > > > > cpp part: > > > > <includes> > > > > <include>**/*.cpp</include> > > > > </includes> > > > > > > > > I wonder if others have noticed these problems? > > > > > > > > -Lari > > > > > > > > On 2024/02/01 19:44:02 Lari Hotari wrote: > > > > > Thanks Nicolo. > > > > > > > > > > > I'd suggest to just disable them at this point > > > > > > > > > > Why would you recommend that? > > > > > > > > > > I wonder what has been causing tests to be skipped in the Maven > > > > > build? In the Junit 5 PR, I have upgraded maven-surefire-plugin and > > > > > tests get run with Junit 5's vintage engine. Is there a known bug in > > > > > older maven plugin versions that tests get skipped without notice? > > > > > > > > > > -Lari > > > > > > > > > > On 2024/02/01 19:38:09 Nicolò Boschi wrote: > > > > > > Hi Lari, > > > > > > They haven't run for years now. > > > > > > > > > > > > When we migrated to Gradle, I made them to work again. See > > > > > > https://lists.apache.org/thread/p9twnlh75kp7t74d8ghmtqbw1f5v4kgk > > > > > > > > > > > > > > > > > > But after switching back to Maven, nobody took care of them and no > > > > > > attempts > > > > > > have been made to make them runnable again > > > > > > > > > > > > I'd suggest to just disable them at this point > > > > > > > > > > > > Nicolò Boschi > > > > > > > > > > > > > > > > > > Il giorno gio 1 feb 2024 alle ore 20:23 Lari Hotari > > > > > > <lhot...@apache.org> ha > > > > > > scritto: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm currently working on a PR to enable writing tests with Junit > > > > > > > 5 and > > > > > > > having Junit 5 tests in addition to Junit 4 tests. This is in PR > > > > > > > https://github.com/apache/bookkeeper/pull/4197 > > > > > > > > > > > > > > Suddenly there are test failures! I first thought that the > > > > > > > upgrade had > > > > > > > broken something, but it turns out that my changes fix problems > > > > > > > and start > > > > > > > running tests that were previously ignored! > > > > > > > > > > > > > > Previously, integration tests ran in 29 seconds! > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > [INFO] BUILD SUCCESS > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > [INFO] Total time: 29.054 s > > > > > > > [INFO] Finished at: 2024-01-31T02:32:17Z > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > Test current server with old clients: > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > [INFO] BUILD SUCCESS > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > [INFO] Total time: 5.464 s > > > > > > > [INFO] Finished at: 2024-01-31T02:28:31Z > > > > > > > [INFO] > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > The docker container tests using arquillian-cube-docker have been > > > > > > > skipped > > > > > > > completely. I managed to get arquillian-cube-docker to run, but > > > > > > > it turns > > > > > > > out that the build isn't properly setup for this. I guess I put > > > > > > > myself in a > > > > > > > rabbit hole... I need some help. > > > > > > > > > > > > > > Could someone share more context about the state of Bookkeeper CI? > > > > > > > Any ongoing effort to improve it? > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > > > > > > > > > >
