Aman Poonia created HBASE-30355:
-----------------------------------
Summary: setupMiniKdc port-conflict retry never triggers because
Kerby throws KrbException, not BindException
Key: HBASE-30355
URL: https://issues.apache.org/jira/browse/HBASE-30355
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 2.6.7, 3.0.0, 4.0.0-alpha-1
Reporter: Aman Poonia
h3. Problem
{{TestLogLevel}} intermittently fails in {{setUp}} when the MiniKDC port is
already in use:
{code}
org.apache.kerby.kerberos.kerb.KrbException: Failed to start
DefaultKrbServer. Address already in use (Bind failed)
at
org.apache.kerby.kerberos.kerb.server.impl.AbstractInternalKdcServer.start(AbstractInternalKdcServer.java:73)
at
org.apache.kerby.kerberos.kerb.server.KdcServer.start(KdcServer.java:268)
at
org.apache.kerby.kerberos.kerb.server.SimpleKdcServer.start(SimpleKdcServer.java:174)
at org.apache.hadoop.minikdc.MiniKdc.start(MiniKdc.java:284)
at
org.apache.hadoop.hbase.http.log.TestLogLevel.setupMiniKdc(TestLogLevel.java:155)
{code}
h3. Root cause
{{setupMiniKdc}} already has retry-on-port-conflict logic, but it only
{{catch (BindException e)}}. The Kerby-backed {{MiniKdc}} does *not* throw
{{java.net.BindException}} — it wraps the bind failure in a Kerby
{{KrbException}}, and (verified) the cause chain carries *no*
{{BindException}}
at all; the port conflict is only visible in the message ("Address already in
use"). As a result {{catch (BindException)}} never matches, the 3x retry is
dead code, and the first port collision fails the test.
The identical block exists in two places:
* {{TestLogLevel.setupMiniKdc}} (hbase-http)
* {{HBaseTestingUtil.setupMiniKdc}} (hbase-server) — affects other secure
MiniKDC tests, e.g. {{TestSecureRESTServer}}, {{TestSecureExport}}.
h3. Fix
Recognise a bind conflict regardless of the wrapper type. Broaden the catch to
{{catch (Exception e)}} and, before retrying, test a predicate that walks the
whole cause chain and also matches the "Address already in use" message;
rethrow anything that is not a bind conflict so real KDC misconfig is never
masked by the retry.
We deliberately do *not* {{catch (BindException | KrbException)}}:
* the discriminator is the message, not the type (the cause chain has no
BindException), and a future MiniKdc/Kerby could use a different wrapper;
* a bare {{catch (KrbException)}} would need the same message check anyway
(a genuine KDC misconfig is also a KrbException);
* it avoids a hard compile-time import of a kerby type, which HBASE-29117
explicitly warns against (kerby version is unpinned and 1.x/2.x are
incompatible across Hadoop versions).
h3. Tests
* {{testKdcBindConflictSurfacesAsKrbException}} — deterministic reproduction:
occupies the KDC port on TCP+UDP, pins MiniKdc to it, and asserts the
failure
is a Kerby {{KrbException}} (not a {{BindException}}) that the predicate
recognises. Reproduces the exact production signature.
* {{testIsBindExceptionRecognizesKerbyWrappedBindFailure}} — unit test for the
predicate: KrbException wrapping a BindException, KrbException with the bind
message only, and a negative (unrelated failure not treated as retryable).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
