Julio J. Gomez Diaz created AVRO-4354:
-----------------------------------------
Summary: [java] Generated equals() (AVRO-3527) throws "Unknown
datum type" for arrays containing logical-type unions, and generated hashCode()
is inconsistent with equals() for CharSequence fields
Key: AVRO-4354
URL: https://issues.apache.org/jira/browse/AVRO-4354
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.12.2, 1.12.1
Environment: h2. Environment
Java 25 (Temurin 25.0.4), Maven 3.9.16, {{avro-maven-plugin}} with its default
configuration, JUnit 5.11.4.
Reporter: Julio J. Gomez Diaz
h2. Summary
Since 1.12.1 the {{SpecificCompiler}} generates {{equals()}} and {{hashCode()}}
for every record (AVRO-3527). Compared to 1.12.0, where records inherited
{{SpecificRecordBase.equals()}} / {{{}hashCode(){}}}, this introduces two
regressions:
# *{{equals()}} throws* {{AvroRuntimeException: Unknown datum type ...}} when
the record has an array whose elements contain a union of {{null}} and a
logical type with a registered conversion ({{{}uuid{}}}, {{{}date{}}},
{{{}timestamp-millis{}}}, ...). It happens both for an _array of records with a
nullable logical-type field_ and for an {_}array of nullable logical-type
values{_}. It is also asymmetric: {{before.equals(after)}} returns
{{{}true{}}}, while {{after.equals(before)}} throws.
# *{{hashCode()}} is inconsistent with {{equals()}}* for {{CharSequence}}
fields (the default {{{}stringType{}}}): a record holding a
{{java.lang.String}} and the same record after a round-trip, which holds an
{{{}org.apache.avro.util.Utf8{}}}, are equal according to {{equals()}} but have
different hash codes. This breaks {{HashSet}} / {{HashMap}} lookups.
Serialization and deserialization are *not* affected. The problems appear when
generated objects are compared or hashed: test assertions,
deduplication/idempotency with {{Set}} / {{{}Map{}}}, {{{}List.contains(){}}},
{{{}Stream.distinct(){}}}, argument matching in mocking libraries, etc. For
duplicates, both decoded objects have the same hash code, so {{HashSet.add()}}
/ {{contains()}} and {{HashMap.get()}} always end up calling {{equals()}} and
always throw.
h2. Affected versions
* *1.12.1* and {*}1.12.2{*}: reproduced (see the results table).
* {{branch-1.12}} (1.12.3-SNAPSHOT) and {{{}main{}}}: not executed. Code
inspection shows the same generated code in {{record.vm}}
({{{}java.util.Objects.equals(this.x, other.x){}}} and {{{}x.hashCode(){}}})
and the same {{{}GenericData.AbstractArray.equals(){}}}.
* {*}1.12.0{*}: not affected, because no {{equals()}} / {{hashCode()}} is
generated.
* Classes generated with the 1.12.0 compiler and run on the 1.12.1 runtime are
*not* affected. The regression is in the generated code, not in the runtime.
h2. How to reproduce
Plain {{avro-maven-plugin}} with its default configuration (only the {{schema}}
goal, {{stringType}} left at its default {{{}CharSequence{}}}):
{code:xml}
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.12.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
</execution>
</executions>
</plugin>
{code}
Schema ({{{}src/main/avro/order.avsc{}}}):
{code:json}
[
{
"type": "record",
"name": "Order",
"namespace": "org.example",
"fields": [
{
"name": "lines",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "Line",
"fields": [
{ "name": "line_id", "type": ["null", { "type": "string",
"logicalType": "uuid" }], "default": null }
]
}
},
"default": []
},
{
"name": "dates",
"type": { "type": "array", "items": ["null", { "type": "int",
"logicalType": "date" }] },
"default": []
}
]
},
{
"type": "record",
"name": "Named",
"namespace": "org.example",
"fields": [
{ "name": "name", "type": "string" }
]
}
]
{code}
Test (JUnit 5):
{code:java}
package org.example;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
import java.time.LocalDate;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.UUID;
import org.junit.jupiter.api.Test;
class GeneratedEqualsTest {
private static final UUID ID =
UUID.fromString("de5714a7-0000-0000-0000-000000000001");
private static Order roundTrip(final Order order) throws IOException {
return Order.getDecoder().decode(Order.getEncoder().encode(order));
}
@Test
void arrayOfRecordsWithNullableUuid() throws IOException {
final Order before = Order.newBuilder()
.setLines(List.of(Line.newBuilder().setLineId(ID).build()))
.setDates(List.of())
.build();
final Order after = roundTrip(before);
assertTrue(before.equals(after)); // passes on 1.12.0 and 1.12.1
assertTrue(after.equals(before)); // 1.12.1:
AvroRuntimeException: Unknown datum type java.util.UUID
assertTrue(after.equals(roundTrip(before))); // 1.12.1: same exception
}
@Test
void arrayOfNullableDate() throws IOException {
final Order before = Order.newBuilder()
.setLines(List.of())
.setDates(List.of(LocalDate.of(2026, 9, 28)))
.build();
final Order after = roundTrip(before);
assertTrue(after.equals(before)); // 1.12.1:
AvroRuntimeException: Unknown datum type java.time.LocalDate
}
@Test
void hashCodeIsConsistentWithEquals() throws IOException {
final Named before = Named.newBuilder().setName("a").build();
// holds a java.lang.String
final Named after =
Named.getDecoder().decode(Named.getEncoder().encode(before)); // holds an
org.apache.avro.util.Utf8
assertTrue(before.equals(after)); // true on 1.12.0 and
1.12.1
assertEquals(before.hashCode(), after.hashCode()); // 1.12.1: fails, equal
objects with different hash codes
final Set<Named> set = new HashSet<>(Set.of(before));
assertTrue(set.contains(after)); // 1.12.1: false
}
}
{code}
h3. Results
||Test||1.12.0||1.12.1||1.12.2 (note 1)||Generator 1.12.0 + runtime 1.12.1||
|arrayOfRecordsWithNullableUuid|pass|{{AvroRuntimeException: Unknown datum type
java.util.UUID}}|same as 1.12.1|pass|
|arrayOfNullableDate|pass|{{AvroRuntimeException: Unknown datum type
java.time.LocalDate}}|same as 1.12.1|pass|
|hashCodeIsConsistentWithEquals|pass|{{AssertionFailedError: expected: <128>
but was: <159>}}|same as 1.12.1|pass|
Note 1: on 1.12.2 the test has to run with
{{{}-Dorg.apache.avro.SERIALIZABLE_PACKAGES=org.example{}}}. Without it, every
decode fails earlier with {{{}SecurityException: Forbidden org.example.Order!
This class is not trusted to be included in Avro schemas ...{}}}, even when
using the generated {{{}getDecoder(){}}}.
Stack trace on 1.12.1:
{noformat}
org.apache.avro.AvroRuntimeException: Unknown datum type java.util.UUID:
de5714a7-0000-0000-0000-000000000001
at
org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:976)
at
org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:935)
at org.apache.avro.generic.GenericData.compare(GenericData.java:1309)
at org.apache.avro.generic.GenericData.compare(GenericData.java:1285)
at org.apache.avro.generic.GenericData.compare(GenericData.java:1299)
at
org.apache.avro.generic.GenericData$AbstractArray.equals(GenericData.java:358)
at org.example.Order.equals(Order.java:373)
at
org.example.GeneratedEqualsTest.arrayOfRecordsWithNullableUuid(GeneratedEqualsTest.java:32)
{noformat}
where {{Order.java:373}} is the generated line:
{code:java}
if (!java.util.Objects.equals(this.lines, other.lines)) {
{code}
h2. Root cause
h3. 1. The equals() exception
The generated {{equals()}} compares non-primitive, non-{{{}CharSequence{}}}
fields with {{java.util.Objects.equals(this.x, other.x)}} (the
{{canGenerateEqualsAndHashCode}} block of {{{}record.vm{}}}). On a decoded
record, the runtime type of an array field is {{{}GenericData.Array{}}}, whose
{{equals()}} is:
{code:java}
public boolean equals(final Object o) {
if (!(o instanceof Collection)) {
return false;
}
return GenericData.get().compare(this, o, this.getSchema(), true) == 0;
}
{code}
{{GenericData.get()}} has no logical-type conversions registered. When
{{compare()}} reaches the union, {{resolveUnion()}} calls
{{{}getSchemaName(){}}}, which cannot classify the {{java.util.UUID}} /
{{java.time.LocalDate}} / {{java.time.Instant}} value and throws.
This defect of {{GenericData.AbstractArray.equals()}} already exists in 1.12.0.
AVRO-4036 reports it for a direct {{List.equals()}} call with a custom logical
type. In 1.12.0 it could only be reached by calling {{equals()}} on the list
directly: {{SpecificRecordBase.equals()}} uses {{{}getSpecificData(){}}}, i.e.
the {{MODEL$}} of the generated class, which carries the conversions, to
compare the whole tree. Since AVRO-3527, *every* generated {{equals()}} of a
record containing such an array goes through {{{}GenericData.Array.equals(){}}}.
The asymmetry comes from the list implementation on each side:
* a {{java.util.List}} built by the user compares element by element:
generated {{{}Line.equals(){}}}, then {{{}UUID.equals(){}}};
* a {{GenericData.Array}} produced by the decoder uses
{{{}GenericData.get(){}}}.
h3. 2. The hashCode() inconsistency
The generated {{hashCode()}} calls {{x.hashCode()}} directly, while the
generated {{equals()}} uses {{Utf8.compareSequences()}} for {{CharSequence}}
fields. {{String.hashCode()}} and {{Utf8.hashCode()}} differ ({{{}"a"{}}}: 97
vs 128), so two objects that are equal according to {{equals()}} get different
hash codes. In 1.12.0, {{SpecificRecordBase.hashCode()}} normalised strings
through {{GenericData.hashCode()}} ({{{}new Utf8(o.toString()).hashCode(){}}}),
so {{equals()}} and {{hashCode()}} were consistent. The same happens with
strings inside collections; see AVRO-4198 for the {{equals()}} side.
h3. 3. No way to opt out
{{SpecificCompiler.canGenerateEqualsAndHashCode(Schema)}} only returns
{{false}} when custom logical type factories are used, so no compiler or
{{avro-maven-plugin}} option can restore the previous behaviour. The only
workaround is a custom {{templateDirectory}} with a copy of {{record.vm}} that
lacks the equals/hashCode block. That copy has to be kept in sync with every
release, because the templates also carry security fixes (e.g. AVRO-4053 /
CVE-2025-33042), and because the compiler runs Velocity in strict mode, so
templates from one version fail with the compiler of another.
h2. Suggested fix
# Generated {{{}equals(){}}}: for non-primitive fields (at least arrays, maps
and unions), delegate to the class model instead of
{{{}java.util.Objects.equals(){}}}, e.g. {{{}MODEL$.compare(this.x, other.x,
SCHEMA$.getFields().get(i).schema(), true) == 0{}}}, or generate an
element-wise comparison. Primitive and {{CharSequence}} fields can keep the
fast path introduced by AVRO-3527.
# Additionally, or alternatively, make {{GenericData.AbstractArray.equals()}}
use the {{GenericData}} instance that created the array (the one that called
{{{}newArray(){}}}) instead of {{{}GenericData.get(){}}}. This would also fix
AVRO-4036.
# Generated {{{}hashCode(){}}}: hash {{CharSequence}} values independently of
their implementation, as {{GenericData.hashCode()}} does, or delegate
non-primitive fields to {{{}MODEL$.hashCode(value, schema){}}}, so that the
result stays consistent with {{{}equals(){}}}.
# Add a compiler / {{avro-maven-plugin}} option (e.g.
{{{}createEqualsAndHashCode{}}}) to disable the generation of {{equals()}} /
{{{}hashCode(){}}}, so users can fall back to {{SpecificRecordBase}} semantics
without forking the templates.
h2. Related issues
* AVRO-3527: introduced the generated {{equals()}} / {{hashCode()}} (cause).
* AVRO-4036: {{GenericData.Array.equals()}} with logical-type unions (same
underlying defect, direct call, 1.12.0).
* AVRO-4198: the generated {{equals()}} returns {{false}} for {{String}} vs
{{Utf8}} in complex types after a round-trip.
* AVRO-4334: {{Utf8.hashCode()}} values changed in 1.12.1.
* AVRO-4183, AVRO-4188: other regressions of the generated methods (fields
named {{result}} / {{{}java{}}}).
* AVRO-4139: equality of arrays of maps, fixed in 1.12.1 (different code path).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
