[
https://issues.apache.org/jira/browse/CASSANDRA-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123946#comment-17123946
]
Berenguer Blasi commented on CASSANDRA-15845:
---------------------------------------------
Hard to repro but seems sometimes we are sending MV mutations on MV creation
before schema agreement and before MV build task:
{noformat}
DEBUG [OptionalTasks:1] 2020-06-02 17:31:36,119 CassandraDaemon.java:389 -
Completed submission of build tasks for any materialized views defined at
startup
INFO [MigrationStage:1] 2020-06-02 17:31:46,145 ColumnFamilyStore.java:849 -
Enqueuing flush of columns: 1.488KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO [Messaging-EventLoop-3-9] 2020-06-02 17:31:46,159 NoSpamLogger.java:91 -
127.0.0.1:7000->127.0.0.2:7000-SMALL_MESSAGES-3d6869c3 incompatible schema
encountered while deserializing a message
org.apache.cassandra.exceptions.UnknownTableException: Couldn't find table with
id 2afc69c0-a4e6-11ea-a44e-7941109e8480. If a table was just created, this is
likely due to the schemanot being fully propagated. Please wait for schema
agreement on table creation.
at
org.apache.cassandra.schema.Schema.getExistingTableMetadata(Schema.java:456)
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:638)
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:380)
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:398)
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:362)
at
org.apache.cassandra.net.Message$Serializer.deserializePost40(Message.java:770)
at
org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:630)
at
org.apache.cassandra.net.InboundMessageHandler.processSmallMessage(InboundMessageHandler.java:320)
at
org.apache.cassandra.net.InboundMessageHandler.processOneContainedMessage(InboundMessageHandler.java:303)
at
org.apache.cassandra.net.InboundMessageHandler.processFrameOfContainedMessages(InboundMessageHandler.java:270)
at
org.apache.cassandra.net.InboundMessageHandler.processIntactFrame(InboundMessageHandler.java:255)
at
org.apache.cassandra.net.InboundMessageHandler.process(InboundMessageHandler.java:246)
at org.apache.cassandra.net.FrameDecoder.deliver(FrameDecoder.java:320)
at
org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:284)
at
org.apache.cassandra.net.FrameDecoder.channelRead(FrameDecoder.java:268)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
.
.
.
ERROR [MutationStage-2] 2020-06-02 17:31:46,414 Keyspace.java:621 - Attempting
to mutate non-existant table 2afc69c0-a4e6-11ea-a44e-7941109e8480 (ks.t_by_v)
ERROR [MutationStage-1] 2020-06-02 17:31:46,414 Keyspace.java:621 - Attempting
to mutate non-existant table 2afc69c0-a4e6-11ea-a44e-7941109e8480 (ks.t_by_v)
ERROR [MutationStage-2] 2020-06-02 17:31:46,415 Keyspace.java:621 - Attempting
to mutate non-existant table 2afc69c0-a4e6-11ea-a44e-7941109e8480 (ks.t_by_v)
ERROR [MutationStage-4] 2020-06-02 17:31:46,415 Keyspace.java:621 - Attempting
to mutate non-existant table 2afc69c0-a4e6-11ea-a44e-7941109e8480 (ks.t_by_v)
ERROR [MutationStage-1] 2020-06-02 17:31:46,415 Keyspace.java:621 - Attempting
to mutate non-existant table 2afc69c0-a4e6-11ea-a44e-7941109e8480 (ks.t_by_v)
.
.
.
INFO [MigrationStage:1] 2020-06-02 17:31:46,417 ColumnFamilyStore.java:398 -
Initializing ks.t_by_v
.
.
.
DEBUG [MigrationStage:1] 2020-06-02 17:31:46,601 ViewBuilder.java:97 - Starting
build of view(ks.t_by_v). Flushing base table ks.t
...
DEBUG [ViewBuildExecutor:1] 2020-06-02 17:31:46,778 ViewBuilderTask.java:127 -
Starting new view build for range (-6661324248839560100,-6533221859438799530]
{noformat}
I have to continue digging.
> dtest-novnode.materialized_views_test.TestMaterializedViews.test_populate_mv_after_insert_wide_rows
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15845
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
> Reporter: Berenguer Blasi
> Assignee: Berenguer Blasi
> Priority: Normal
> Fix For: 4.0-beta
>
>
> See failure
> [here|https://ci-cassandra.apache.org/job/Cassandra-trunk/155/testReport/dtest-novnode.materialized_views_test/TestMaterializedViews/test_populate_mv_after_insert_wide_rows_2/]
> {quote}Error Message
> assert 160 == 0 + where 160 = Row(count=160).count
> Stacktrace
> self = <materialized_views_test.TestMaterializedViews object at
> 0x7fcb232b5760>
> def test_populate_mv_after_insert_wide_rows(self):
> """Test that a view is OK when created with existing data with wide
> rows"""
> session = self.prepare(consistency_level=ConsistencyLevel.QUORUM)
>
> session.execute("CREATE TABLE t (id int, v int, PRIMARY KEY (id, v))")
>
> for i in range(5):
> for j in range(10000):
> session.execute("INSERT INTO t (id, v) VALUES ({},
> {})".format(i, j))
>
> session.execute(("CREATE MATERIALIZED VIEW t_by_v AS SELECT * FROM t
> WHERE v IS NOT NULL "
> "AND id IS NOT NULL PRIMARY KEY (v, id)"))
>
> logger.debug("wait for view to build")
> self._wait_for_view("ks", "t_by_v")
>
> logger.debug("wait that all batchlogs are replayed")
> > self._replay_batchlogs()
> materialized_views_test.py:350:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self = <materialized_views_test.TestMaterializedViews object at
> 0x7fcb232b5760>
> def _replay_batchlogs(self):
> for node in self.cluster.nodelist():
> if node.is_running():
> logger.debug("Replaying batchlog on node
> {}".format(node.name))
> node.nodetool("replaybatchlog")
> # CASSANDRA-13069 - Ensure replayed mutations are removed
> from the batchlog
> node_session = self.patient_exclusive_cql_connection(node)
> result = list(node_session.execute("SELECT count(*) FROM
> system.batches;"))
> > assert result[0].count == 0
> E assert 160 == 0
> E + where 160 = Row(count=160).count
> materialized_views_test.py:171: AssertionError{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
