[
https://issues.apache.org/jira/browse/CASSANDRA-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601741#comment-17601741
]
Andres de la Peña commented on CASSANDRA-17872:
-----------------------------------------------
Great! It seems that the output of each runner shows two port conflicts with a
single retry:
{code}
Runner 0
20:12:00,696 tools.jmxutils INFO Port 8778 in use, trying again on 8487
20:12:42,11 tools.jmxutils INFO Port 8778 in use, trying again on 8897
Runner 1
20:11:26,738 tools.jmxutils INFO Port 8778 in use, trying again on 8276
20:12:07,142 tools.jmxutils INFO Port 8778 in use, trying again on 8723
Runner 2
20:11:06,398 tools.jmxutils INFO Port 8778 in use, trying again on 8464
20:11:53,493 tools.jmxutils INFO Port 8778 in use, trying again on 8897
Runner 3
20:09:40,106 tools.jmxutils INFO Port 8778 in use, trying again on 8569
20:10:18,843 tools.jmxutils INFO Port 8778 in use, trying again on 8476
Runner 4
20:08:12,282 tools.jmxutils INFO Port 8778 in use, trying again on 8569
20:08:55,857 tools.jmxutils INFO Port 8778 in use, trying again on 8084
Runner 5
20:11:11,272 tools.jmxutils INFO Port 8778 in use, trying again on 8441
20:11:56,422 tools.jmxutils INFO Port 8778 in use, trying again on 8434
Runner 6
20:09:32,74 tools.jmxutils INFO Port 8778 in use, trying again on 8759
20:10:15,830 tools.jmxutils INFO Port 8778 in use, trying again on 8201
Runner 7
20:09:02,106 tools.jmxutils INFO Port 8778 in use, trying again on 8953
20:09:42,609 tools.jmxutils INFO Port 8778 in use, trying again on 8408
Runner 8
20:10:02,404 tools.jmxutils INFO Port 8778 in use, trying again on 8754
20:10:41,701 tools.jmxutils INFO Port 8778 in use, trying again on 8449
Runner 9
20:10:49,294 tools.jmxutils INFO Port 8778 in use, trying again on 8377
20:12:06,64 tools.jmxutils INFO Port 8778 in use, trying again on 8940
Runner 10
20:10:49,294 tools.jmxutils INFO Port 8778 in use, trying again on 8377
20:12:06,64 tools.jmxutils INFO Port 8778 in use, trying again on 8940
Runner 11
20:10:01,299 tools.jmxutils INFO Port 8778 in use, trying again on 8124
20:10:41,876 tools.jmxutils INFO Port 8778 in use, trying again on 8843
Runner 12
20:10:35,143 tools.jmxutils INFO Port 8778 in use, trying again on 8218
20:11:14,32 tools.jmxutils INFO Port 8778 in use, trying again on 8915
Runner 13
20:08:55,909 tools.jmxutils INFO Port 8778 in use, trying again on 8645
20:09:35,366 tools.jmxutils INFO Port 8778 in use, trying again on 8764
Runner 14
20:11:14,633 tools.jmxutils INFO Port 8778 in use, trying again on 8981
20:12:01,516 tools.jmxutils INFO Port 8778 in use, trying again on 8631
Runner 15
20:10:53,581 tools.jmxutils INFO Port 8778 in use, trying again on 8759
20:11:36,26 tools.jmxutils INFO Port 8778 in use, trying again on 8206
Runner 16
20:09:21,359 tools.jmxutils INFO Port 8778 in use, trying again on 8884
20:10:01,638 tools.jmxutils INFO Port 8778 in use, trying again on 8206
Runner 17
20:08:36,993 tools.jmxutils INFO Port 8778 in use, trying again on 8515
20:09:15,834 tools.jmxutils INFO Port 8778 in use, trying again on 8637
Runner 18
20:09:47,922 tools.jmxutils INFO Port 8778 in use, trying again on 8754
20:10:35,190 tools.jmxutils INFO Port 8778 in use, trying again on 8554
Runner 19
20:10:49,276 tools.jmxutils INFO Port 8778 in use, trying again on 8884
20:11:31,124 tools.jmxutils INFO Port 8778 in use, trying again on 8198
Runner 20
20:10:25,352 tools.jmxutils INFO Port 8778 in use, trying again on 8540
20:11:05,916 tools.jmxutils INFO Port 8778 in use, trying again on 8433
Runner 21
20:10:11,926 tools.jmxutils INFO Port 8778 in use, trying again on 8658
20:10:52,447 tools.jmxutils INFO Port 8778 in use, trying again on 8049
Runner 22
20:10:28,433 tools.jmxutils INFO Port 8778 in use, trying again on 8953
20:11:11,215 tools.jmxutils INFO Port 8778 in use, trying again on 8408
Runner 23
20:10:57,540 tools.jmxutils INFO Port 8778 in use, trying again on 8594
20:11:36,451 tools.jmxutils INFO Port 8778 in use, trying again on 8843
Runner 24
20:10:51,980 tools.jmxutils INFO Port 8778 in use, trying again on 8198
20:11:33,979 tools.jmxutils INFO Port 8778 in use, trying again on 8206
{code}
> Dtests failing intermittently on Jolokia agent
> ----------------------------------------------
>
> Key: CASSANDRA-17872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17872
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
> Reporter: Andres de la Peña
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 4.x
>
>
> Some apparently unrealeted Python dtests fail with an output of the form:
> {code:java}
> Error Message
> subprocess.CalledProcessError: Command
> '('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '-cp',
> '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandra/cassandra/cassandra-dtest/tools/../lib/jolokia-jvm-1.7.1-agent.jar',
> 'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1', 'start',
> '706')' returned non-zero exit status 1.
> Stacktrace
> self = <auth_test.TestAuthRoles object at 0x7fc6cb4313a0>
> (...)
>
> mbean = make_mbean('auth', type='RolesCache')
> > with JolokiaAgent(self.cluster.nodelist()[0]) as jmx:
> auth_test.py:1888:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> tools/jmxutils.py:309: in __enter__
> self.start()
> tools/jmxutils.py:187: in start
> subprocess.check_output(args, stderr=subprocess.STDOUT)
> /usr/lib/python3.8/subprocess.py:415: in check_output
> return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> input = None, capture_output = False, timeout = None, check = True
> popenargs = (('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '-cp',
> '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandr...t/tools/../lib/jolokia-jvm-1.7.1-agent.jar',
> 'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1', ...),)
> kwargs = {'stderr': -2, 'stdout': -1}
> process = <subprocess.Popen object at 0x7fc6c9afb910>
> stdout = b"Couldn't start agent for PID 706\nPossible reason could be that
> port '8778' is already occupied.\nPlease check the standard output of the
> target process for a detailed error message.\n"
> stderr = None, retcode = 1
> (...)
> if check and retcode:
> > raise CalledProcessError(retcode, process.args,
> output=stdout, stderr=stderr)
> E subprocess.CalledProcessError: Command
> '('/usr/lib/jvm/java-8-openjdk-amd64/bin/java', '-cp',
> '/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/cassandra/cassandra/cassandra-dtest/tools/../lib/jolokia-jvm-1.7.1-agent.jar',
> 'org.jolokia.jvmagent.client.AgentLauncher', '--host', '127.0.0.1', 'start',
> '706')' returned non-zero exit status 1.
> /usr/lib/python3.8/subprocess.py:516: CalledProcessError
> {code}
> Here is a of bunch hits in different tests across multiple branches:
> *
> [https://app.circleci.com/pipelines/github/adelapena/cassandra/2035/workflows/1e06bd6d-8bd6-4703-85db-2b41e964134e/jobs/20403]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-3.11/387/testReport/dtest-novnode.thrift_hsha_test/TestThriftHSHA/test_closing_connections/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/454/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairLegacyStreaming/test_transient_incremental_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/461/testReport/dtest-novnode.read_repair_test/TestSpeculativeReadRepair/test_failed_read_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/461/testReport/dtest-novnode.transient_replication_test/TestTransientReplication/test_cheap_quorums/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/464/testReport/dtest-offheap.repair_tests.incremental_repair_test/TestIncRepair/test_parent_repair_session_cleanup/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/465/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairLegacyStreaming/test_transient_incremental_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.0/465/testReport/dtest-offheap.repair_tests.incremental_repair_test/TestIncRepair/test_repaired_tracking_with_partition_deletes/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/135/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairStreamEntireSSTable/test_primary_range_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/135/testReport/dtest.auth_test/TestNetworkAuth/test_revoked_login/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/145/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairLegacyStreaming/test_primary_range_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/148/testReport/dtest-novnode.auth_test/TestAuthRoles/test_role_caching_authenticated_user/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/151/testReport/dtest-novnode.read_repair_test/TestSpeculativeReadRepair/test_speculative_data_request/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-4.1/151/testReport/dtest.read_repair_test/TestSpeculativeReadRepair/test_quorum_requirement_on_speculated_read/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1288/testReport/dtest.jmx_test/TestJMX/test_mv_metric_mbeans_release/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1295/testReport/dtest-novnode.client_request_metrics_local_remote_test/TestClientRequestMetricsLocalRemote/test_paxos/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1295/testReport/dtest-offheap.read_repair_test/TestSpeculativeReadRepair/test_quorum_requirement/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1296/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairStreamEntireSSTable/test_speculative_write_repair_cycle/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1296/testReport/dtest-offheap.configuration_test/TestConfiguration/test_change_durable_writes/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1300/testReport/dtest-novnode.read_repair_test/TestSpeculativeReadRepair/test_failed_read_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1300/testReport/dtest-novnode.transient_replication_test/TestTransientReplicationRepairStreamEntireSSTable/test_optimized_primary_range_repair/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1301/testReport/dtest-novnode.client_request_metrics_local_remote_test/TestClientRequestMetricsLocalRemote/test_batch_and_slice/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1301/testReport/dtest-novnode.client_request_metrics_local_remote_test/TestClientRequestMetricsLocalRemote/test_write_and_read/]
> *
> [https://ci-cassandra.apache.org/job/Cassandra-trunk/1302/testReport/dtest-upgrade.upgrade_tests.regression_test/TestForRegressionsUpgrade_current_3_11_x_To_indev_trunk/test13294/]
> Note the common {{with JolokiaAgent(self.cluster.nodelist()[0])}} and
> {{"Possible reason could be that port '8778' is already occupied."}} parts.
> So far, the issue doesn't seem to reproduce on 3.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
