Hi Paul,
I have been looking at this issue for the past few days. Thanks for this
valuable information.
When run parallelly on an x86 machine and an s390x machine I was able to see
that an encoding to byte string varied at an encode function call from
pylsqpack.
Since it was a wrapper to a C function PDB was not able to step into that
particular function call and see what is happening under the hood.
I am still checking for ways to debug that properly.
Thanks
Pranav
________________________________
From: Paul Gevers
Sent: Sunday, March 9, 2025 9:33 PM
To: sub...@bugs.debian.org
Subject: [EXTERNAL] Bug#1099935: dnspython: autopkgtest regression on s390x:
bad request
Source: dnspython
Version: 2.7.0-1
Severity: serious
User: debian...@lists.debian.org
Usertags: regression
User: debian-s...@lists.debian.org
Usertags: s390x
X-Debbugs-CC: debian-s...@lists.debian.org
Dear maintainer(s),
With a recent upload of dnspython the autopkgtest of dnspython fails in
testing when that autopkgtest is run with the binary packages of
dnspython from unstable on s390x. It passes when run with only packages
from testing. In tabular form:
pass fail
dnspython from testing 2.7.0-1
all others from testing from testing
I copied some of the output at the bottom of this report.
Currently this regression is blocking the migration to testing [1]. Can
you please investigate the situation and fix it?
More information about this bug and the reason for filing it can be found on
https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformation
Paul
[1] https://qa.debian.org/excuses.php?package=dnspython
https://ci.debian.net/data/autopkgtest/testing/s390x/d/dnspython/58550446/log.gz
=================================== FAILURES
===================================
120s ________________________ AsyncTests.testDoH3GetRequest
_________________________
120s 120s fut = <coroutine object AsyncioQuicStream._wait_for_wake_up at
0x3ffaa7cd700>
120s timeout = 3.977640151977539
120s 120s async def wait_for(fut, timeout):
120s """Wait for the single Future or coroutine to complete,
with timeout.
120s 120s Coroutine will be wrapped in Task.
120s 120s Returns result of the Future or coroutine. When a
timeout occurs,
120s it cancels the task and raises TimeoutError. To avoid the task
120s cancellation, wrap it in shield().
120s 120s If the wait is cancelled, the task is also cancelled.
120s 120s If the task suppresses the cancellation and
returns a value instead,
120s that value is returned.
120s 120s This function is a coroutine.
120s """
120s # The special case for timeout <= 0 is for the following case:
120s #
120s # async def test_waitfor():
120s # func_started = False
120s #
120s # async def func():
120s # nonlocal func_started
120s # func_started = True
120s #
120s # try:
120s # await asyncio.wait_for(func(), 0)
120s # except asyncio.TimeoutError:
120s # assert not func_started
120s # else:
120s # assert False
120s #
120s # asyncio.run(test_waitfor())
120s 120s 120s if timeout is not None and timeout <= 0:
120s fut = ensure_future(fut)
120s 120s if fut.done():
120s return fut.result()
120s 120s await _cancel_and_wait(fut)
120s try:
120s return fut.result()
120s except exceptions.CancelledError as exc:
120s raise TimeoutError from exc
120s 120s async with timeouts.timeout(timeout):
120s > return await fut
120s 120s /usr/lib/python3.13/asyncio/tasks.py:507: 120s _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 120s
/usr/lib/python3/dist-packages/dns/quic/_asyncio.py:32: in _wait_for_wake_up
120s await self._wake_up.wait()
120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ 120s 120s self = <asyncio.locks.Condition object at
0x3ffaa8b2d50 [unlocked]>
120s 120s async def wait(self):
120s """Wait until notified.
120s 120s If the calling task has not acquired the lock when
this
120s method is called, a RuntimeError is raised.
120s 120s This method releases the underlying lock, and then
blocks
120s until it is awakened by a notify() or notify_all() call for
120s the same condition variable in another task. Once
120s awakened, it re-acquires the lock and returns True.
120s 120s This method may return spuriously,
120s which is why the caller should always
120s re-check the state and be prepared to wait() again.
120s """
120s if not self.locked():
120s raise RuntimeError('cannot wait on un-acquired lock')
120s 120s fut = self._get_loop().create_future()
120s self.release()
120s try:
120s try:
120s self._waiters.append(fut)
120s try:
120s > await fut
120s E asyncio.exceptions.CancelledError
120s 120s /usr/lib/python3.13/asyncio/locks.py:272: CancelledError
120s 120s The above exception was the direct cause of the following
exception:
120s 120s self = <dns.quic._asyncio.AsyncioQuicStream object at
0x3ffaa82fcb0>
120s expiration = 1741478723.9806786
120s 120s async def wait_for_end(self, expiration):
120s while True:
120s timeout = self._timeout_from_expiration(expiration)
120s if self._buffer.seen_end():
120s return
120s try:
120s > await asyncio.wait_for(self._wait_for_wake_up(),
timeout)
120s 120s /usr/lib/python3/dist-packages/dns/quic/_asyncio.py:52: 120s _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ 120s /usr/lib/python3.13/asyncio/tasks.py:506: in wait_for
120s async with timeouts.timeout(timeout):
120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ 120s 120s self = <Timeout [expired]>
120s exc_type = <class 'asyncio.exceptions.CancelledError'>
120s exc_val = CancelledError(), exc_tb = <traceback object at
0x3ffaa5f3a40>
120s 120s async def __aexit__(
120s self,
120s exc_type: Optional[Type[BaseException]],
120s exc_val: Optional[BaseException],
120s exc_tb: Optional[TracebackType],
120s ) -> Optional[bool]:
120s assert self._state in (_State.ENTERED, _State.EXPIRING)
120s 120s if self._timeout_handler is not None:
120s self._timeout_handler.cancel()
120s self._timeout_handler = None
120s 120s if self._state is _State.EXPIRING:
120s self._state = _State.EXPIRED
120s 120s if self._task.uncancel() <= self._cancelling
and exc_type is not None:
120s # Since there are no new cancel requests, we're
120s # handling this.
120s if issubclass(exc_type, exceptions.CancelledError):
120s > raise TimeoutError from exc_val
120s E TimeoutError
120s 120s /usr/lib/python3.13/asyncio/timeouts.py:116: TimeoutError
120s 120s During handling of the above exception, another exception
occurred:
120s 120s self = <tests.test_async.AsyncTests testMethod=testDoH3GetRequest>
120s 120s @unittest.skipIf(not dns.quic.have_quic, "aioquic not
available")
120s def testDoH3GetRequest(self):
120s async def run():
120s nameserver_url =
random.choice(KNOWN_ANYCAST_DOH3_RESOLVER_URLS)
120s q = dns.message.make_query("dns.google.", dns.rdatatype.A)
120s r = await dns.asyncquery.https(
120s q,
120s nameserver_url,
120s post=False,
120s timeout=4,
120s family=family,
120s http_version=dns.asyncquery.HTTPVersion.H3,
120s )
120s self.assertTrue(q.is_response(r))
120s 120s > self.async_run(run)
120s 120s tests/test_async.py:577: 120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 120s
tests/test_async.py:188: in async_run
120s return asyncio.run(afunc())
120s /usr/lib/python3.13/asyncio/runners.py:195: in run
120s return runner.run(main)
120s /usr/lib/python3.13/asyncio/runners.py:118: in run
120s return self._loop.run_until_complete(task)
120s /usr/lib/python3.13/asyncio/base_events.py:725: in run_until_complete
120s return future.result()
120s tests/test_async.py:567: in run
120s r = await dns.asyncquery.https(
120s /usr/lib/python3/dist-packages/dns/asyncquery.py:583: in https
120s return await _http3(
120s /usr/lib/python3/dist-packages/dns/asyncquery.py:724: in _http3
120s wire = await stream.receive(_remaining(expiration))
120s /usr/lib/python3/dist-packages/dns/quic/_asyncio.py:59: in receive
120s await self.wait_for_end(expiration)
120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ 120s 120s self = <dns.quic._asyncio.AsyncioQuicStream object
at 0x3ffaa82fcb0>
120s expiration = 1741478723.9806786
120s 120s async def wait_for_end(self, expiration):
120s while True:
120s timeout = self._timeout_from_expiration(expiration)
120s if self._buffer.seen_end():
120s return
120s try:
120s await asyncio.wait_for(self._wait_for_wake_up(),
timeout)
120s except TimeoutError:
120s > raise dns.exception.Timeout
120s E dns.exception.Timeout: The DNS operation timed out.
120s 120s /usr/lib/python3/dist-packages/dns/quic/_asyncio.py:54: Timeout
120s ______________________ TrioAsyncTests.testDoH3GetRequest
_______________________
120s + Exception Group Traceback (most recent call last):
120s | File "/usr/lib/python3.13/unittest/case.py", line 58, in
testPartExecutor
120s | yield
120s | File "/usr/lib/python3.13/unittest/case.py", line 651, in run
120s | self._callTestMethod(testMethod)
120s | ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
120s | File "/usr/lib/python3.13/unittest/case.py", line 606, in
_callTestMethod
120s | if method() is not None:
120s | ~~~~~~^^
120s | File
"/tmp/autopkgtest-lxc.ne7u8yhf/downtmp/autopkgtest_tmp/tests/test_async.py",
line 577, in testDoH3GetRequest
120s | self.async_run(run)
120s | ~~~~~~~~~~~~~~^^^^^
120s | File
"/tmp/autopkgtest-lxc.ne7u8yhf/downtmp/autopkgtest_tmp/tests/test_async.py",
line 719, in async_run
120s | return trio.run(afunc)
120s | ~~~~~~~~^^^^^^^
120s | File "/usr/lib/python3/dist-packages/trio/_core/_run.py",
line 2407, in run
120s | raise runner.main_task_outcome.error
120s | File
"/tmp/autopkgtest-lxc.ne7u8yhf/downtmp/autopkgtest_tmp/tests/test_async.py",
line 567, in run
120s | r = await dns.asyncquery.https(
120s | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
120s | ...<6 lines>...
120s | )
120s | ^
120s | File "/usr/lib/python3/dist-packages/dns/asyncquery.py", line
583, in https
120s | return await _http3(
120s | ^^^^^^^^^^^^^
120s | ...<11 lines>...
120s | )
120s | ^
120s | File "/usr/lib/python3/dist-packages/dns/asyncquery.py", line
714, in _http3
120s | async with cfactory() as context:
120s | ~~~~~~~~^^
120s | File "/usr/lib/python3/dist-packages/trio/_core/_run.py",
line 1039, in __aexit__
120s | raise combined_error_from_nursery
120s | ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
120s +-+---------------- 1 ----------------
120s | Traceback (most recent call last):
120s | File "/usr/lib/python3/dist-packages/dns/asyncquery.py",
line 724, in _http3
120s | wire = await stream.receive(_remaining(expiration))
120s | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
120s | File "/usr/lib/python3/dist-packages/dns/quic/_trio.py",
line 60, in receive
120s | raise dns.exception.Timeout
120s | dns.exception.Timeout: The DNS operation timed out.
120s +------------------------------------
120s ___________________ DNSOverHTTP3TestCase.testDoH3GetRequest
____________________
120s 120s self = <tests.test_doh.DNSOverHTTP3TestCase
testMethod=testDoH3GetRequest>
120s 120s def testDoH3GetRequest(self):
120s nameserver_url =
random.choice(KNOWN_ANYCAST_DOH3_RESOLVER_URLS)
120s q = dns.message.make_query("dns.google.", dns.rdatatype.A)
120s > r = dns.query.https(
120s q,
120s nameserver_url,
120s post=False,
120s timeout=4,
120s family=family,
120s http_version=dns.query.HTTPVersion.H3,
120s )
120s 120s tests/test_doh.py:200: 120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 120s
/usr/lib/python3/dist-packages/dns/query.py:472: in https
120s return _http3(
120s /usr/lib/python3/dist-packages/dns/query.py:629: in _http3
120s _check_status(stream.headers(), where, wire)
120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ 120s 120s headers = [(b':status', b'400'), (b'content-type',
b'text/html; charset=UTF-8'), (b'referrer-policy', b'no-referrer'),
(b'content-length', b'1555'), (b'date', b'Sun, 09 Mar 2025 00:05:44 GMT')]
120s peer = '8.8.8.8'
120s wire = b'<!DOCTYPE html>\n<html lang=en>\n <meta charset=utf-8>\n
<meta name=viewport content="initial-scale=1,
minimum-sca...error.</ins>\n <p>Your client has issued a malformed or
illegal request. <ins>That\xe2\x80\x99s all we know.</ins>\n'
120s 120s def _check_status(headers: dns.quic.Headers, peer: str,
wire: bytes) -> None:
120s value = _find_header(headers, b":status")
120s if value is None:
120s raise SyntaxError("no :status header in response")
120s status = int(value)
120s if status < 0:
120s raise SyntaxError("status is negative")
120s if status < 200 or status > 299:
120s error = ""
120s if len(wire) > 0:
120s try:
120s error = ": " + wire.decode()
120s except Exception:
120s pass
120s > raise ValueError(f"{peer} responded with status code
{status}{error}")
120s E ValueError: 8.8.8.8 responded with status code 400:
<!DOCTYPE html>
120s E <html lang=en>
120s E <meta charset=utf-8>
120s E <meta name=viewport content="initial-scale=1,
minimum-scale=1, width=device-width">
120s E <title>Error 400 (Bad Request)!!1</title>
120s E <style>
120s E *{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a
img{border:0}@media screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
120s E </style>
120s E <a href=//www.google.com/><span id=logo
aria-label=Google></span></a>
120s E <p><b>400.</b> <ins>That’s an error.</ins>
120s E <p>Your client has issued a malformed or illegal
request. <ins>That’s all we know.</ins>
120s 120s /usr/lib/python3/dist-packages/dns/query.py:592: ValueError
120s _________________ DNSOverHTTP3TestCase.test_build_url_from_ip
__________________
120s 120s self = <tests.test_doh.DNSOverHTTP3TestCase
testMethod=test_build_url_from_ip>
120s 120s def test_build_url_from_ip(self):
120s self.assertTrue(resolver_v4_addresses or resolver_v6_addresses)
120s if resolver_v4_addresses:
120s nameserver_ip = random.choice(resolver_v4_addresses)
120s q = dns.message.make_query("example.com.", dns.rdatatype.A)
120s # For some reason Google's DNS over HTTPS fails when
you POST to
120s # https://8.8.8.8/dns-query
120s # So we're just going to do GET requests here
120s > r = dns.query.https(
120s q,
120s nameserver_ip,
120s post=False,
120s timeout=4,
120s http_version=dns.query.HTTPVersion.H3,
120s )
120s 120s tests/test_doh.py:231: 120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 120s
/usr/lib/python3/dist-packages/dns/query.py:472: in https
120s return _http3(
120s /usr/lib/python3/dist-packages/dns/query.py:629: in _http3
120s _check_status(stream.headers(), where, wire)
120s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ 120s 120s headers = [(b':status', b'400'), (b'content-type',
b'text/html; charset=UTF-8'), (b'referrer-policy', b'no-referrer'),
(b'content-length', b'1555'), (b'date', b'Sun, 09 Mar 2025 00:05:45 GMT')]
120s peer = '8.8.8.8'
120s wire = b'<!DOCTYPE html>\n<html lang=en>\n <meta charset=utf-8>\n
<meta name=viewport content="initial-scale=1,
minimum-sca...error.</ins>\n <p>Your client has issued a malformed or
illegal request. <ins>That\xe2\x80\x99s all we know.</ins>\n'
120s 120s def _check_status(headers: dns.quic.Headers, peer: str,
wire: bytes) -> None:
120s value = _find_header(headers, b":status")
120s if value is None:
120s raise SyntaxError("no :status header in response")
120s status = int(value)
120s if status < 0:
120s raise SyntaxError("status is negative")
120s if status < 200 or status > 299:
120s error = ""
120s if len(wire) > 0:
120s try:
120s error = ": " + wire.decode()
120s except Exception:
120s pass
120s > raise ValueError(f"{peer} responded with status code
{status}{error}")
120s E ValueError: 8.8.8.8 responded with status code 400:
<!DOCTYPE html>
120s E <html lang=en>
120s E <meta charset=utf-8>
120s E <meta name=viewport content="initial-scale=1,
minimum-scale=1, width=device-width">
120s E <title>Error 400 (Bad Request)!!1</title>
120s E <style>
120s E *{margin:0;padding:0}html,code{font:15px/22px
arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7%
auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* >
body{background:url(//www.google.com/images/errors/robot.png) 100% 5px
no-repeat;padding-right:205px}p{margin:11px 0
22px;overflow:hidden}ins{color:#777;text-decoration:none}a
img{border:0}@media screen and
(max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png)
no-repeat;margin-left:-5px}@media only screen and
(min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat 0% 0%/100%
100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
0}}@media only screen and
(-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png)
no-repeat;-webkit-background-size:100%
100%}}#logo{display:inline-block;height:54px;width:150px}
120s E </style>
120s E <a href=//www.google.com/><span id=logo
aria-label=Google></span></a>
120s E <p><b>400.</b> <ins>That’s an error.</ins>
120s E <p>Your client has issued a malformed or illegal
request. <ins>That’s all we know.</ins>
120s 120s /usr/lib/python3/dist-packages/dns/query.py:592: ValueError
120s =========================== short test summary info
============================
120s FAILED tests/test_async.py::AsyncTests::testDoH3GetRequest -
dns.exception.Ti...
120s FAILED tests/test_async.py::TrioAsyncTests::testDoH3GetRequest -
ExceptionGro...
120s FAILED tests/test_doh.py::DNSOverHTTP3TestCase::testDoH3GetRequest
- ValueErr...
120s FAILED
tests/test_doh.py::DNSOverHTTP3TestCase::test_build_url_from_ip - Valu...
120s ======================= 4 failed, 1356 passed in 45.08s
========================
121s autopkgtest [00:06:04]: test py3
