Hi, Vinay, thank you for diagnosing this.
+1 on the release then since this a test bug and 13004 has already extensively litigated the UX of this. Ariel On Thu, Feb 7, 2019, at 3:49 AM, Vinay Chella wrote: > Hi Ariel, > > test_simple_bootstrap_mixed_versions issue is related to CASSANDRA-13004 > <https://issues.apache.org/jira/browse/CASSANDRA-13004>, which introduced > "cassandra.force_3_0_protocol_version" for schema migrations during > upgrades from 3.0.14 upwards. This flag is missing in > `test_simple_bootstrap_mixed_versions` upgrade test while we are > adding/bootstrapping 3.11.4 node to an existing 3.5 version of C* node. > This resulted in `ks` keyspace schema/data not being bootstrapped to the > new node. > > I debugged and confirmed that MigrationManager::is30Compatible > <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L181-L185> > is returning false which is forcing > MigrationManager::shouldPullSchemaFrom > <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L168-L177> > to return false as well. > > *From debug logs:* > DEBUG [GossipStage:1] 2019-02-06 23:20:47,392 MigrationManager.java:115 - > Not pulling schema because versions match or shouldPullSchemaFrom returned > false > > Here is the updated dtest branch: > https://github.com/vinaykumarchella/cassandra-dtest/tree/fix_failing_upgradetest > > dtests on CircleCI: https://circleci.com/gh/vinaykumarchella/cassandra/345 > > P.S: While MigrationManager > <https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/MigrationManager.java#L181-L185> > confirms that schema migrations from pre 3.11 are not allowed without > `cassandra.force_3_0_protocol_version` option, release notes for 3.11 > are > confusing - docs > <https://github.com/apache/cassandra/blob/cassandra-3.11/NEWS.txt#L173-L174> > > Let me know if this looks good to you, I will send a patch to > cassandra-dtest > > > > Thanks, > Vinay Chella > > > On Wed, Feb 6, 2019 at 8:07 PM Vinay Chella <vinaykumar...@gmail.com> wrote: > > > Hi Ariel, > > > > Sure, I am volunteering to debug this. Will update the progress here. > > > > Thanks, > > Vinay > > > > > > On Wed, Feb 6, 2019 at 1:41 PM Ariel Weisberg <ar...@weisberg.ws> wrote: > > > >> Hi, > >> > >> It fails consistently. I don't know why the data is not evenly > >> distributed. Can someone volunteer to debug this failing test to make sure > >> there isn't an issue with bootstrap in 3.11? > >> > >> https://circleci.com/gh/aweisberg/cassandra/2593 > >> > >> Thanks, > >> Ariel > >> On Wed, Feb 6, 2019, at 3:11 PM, Ariel Weisberg wrote: > >> > Hi, > >> > > >> > -0 > >> > > >> > bootstrap_upgrade_test.py test_simple_bootstrap_mixed_versions fails > >> > because it doesn't see the expected on disk size within 30% of the > >> > expected value. It's bootstrapping a new version node and runs cleanup > >> > on the existing node. If the data were evenly distributed the on disk > >> > size should be similar. > >> > > >> > https://circleci.com/gh/aweisberg/cassandra/2591#tests/containers/40 > >> > > >> > I don't have time to see if this reproduces manually. I kicked off the > >> > tests again to see if reproduces. > >> > https://circleci.com/gh/aweisberg/cassandra/2593 > >> > > >> > Ariel > >> > > >> > On Wed, Feb 6, 2019, at 5:02 AM, Marcus Eriksson wrote: > >> > > +1 > >> > > > >> > > Den ons 6 feb. 2019 kl 10:52 skrev Benedict Elliott Smith < > >> > > bened...@apache.org>: > >> > > > >> > > > +1 > >> > > > > >> > > > > On 6 Feb 2019, at 08:01, Tommy Stendahl < > >> tommy.stend...@ericsson.com> > >> > > > wrote: > >> > > > > > >> > > > > +1 (non-binding) > >> > > > > > >> > > > > /Tommy > >> > > > > > >> > > > > On lör, 2019-02-02 at 18:31 -0600, Michael Shuler wrote: > >> > > > > > >> > > > > I propose the following artifacts for release as 3.11.4. > >> > > > > > >> > > > > sha1: fd47391aae13bcf4ee995abcde1b0e180372d193 > >> > > > > Git: > >> > > > > > >> > > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.4-tentative > >> > > > > Artifacts: > >> > > > > > >> > > > > >> https://repository.apache.org/content/repositories/orgapachecassandra-1170/org/apache/cassandra/apache-cassandra/3.11.4/ > >> > > > > Staging repository: > >> > > > > > >> > > > > >> https://repository.apache.org/content/repositories/orgapachecassandra-1170/ > >> > > > > > >> > > > > The Debian and RPM packages are available here: > >> > > > > http://people.apache.org/~mshuler > >> > > > > > >> > > > > The vote will be open for 72 hours (longer if needed). > >> > > > > > >> > > > > [1]: CHANGES.txt: > >> > > > > > >> > > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative > >> > > > > [2]: NEWS.txt: > >> > > > > > >> > > > > >> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative > >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- > >> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > > > >> > > > > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> > For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
