linliu-code opened a new pull request, #20105:
URL: https://github.com/apache/hudi/pull/20105
### Describe the issue this Pull Request addresses
Addresses the equality defect reported in #20104.
`HoodieFileIndex` is a case class, so Scala derives `equals`/`hashCode` from
every constructor
parameter — `fileStatusCache` included. `@transient` affects serialization,
not equality.
`FileStatusCache.getOrCreate(session)` returns a fresh `SharedInMemoryCache`
view on every call
and the class has no `equals` of its own, so two indexes over the same table
never compared equal.
Spark's `CacheManager` matches entries by plan. With the indexes unequal,
`recacheByPlan` could
not find the entry already cached for a table once `refreshTable` rebuilt
its relation, so the
old materialised dataset was stranded rather than replaced. Observed
behaviour before this
change, five `CACHE TABLE` / `INSERT` cycles on one table:
| cycle | Hudi entries | Parquet entries |
| --- | --- | --- |
| 1 | 1 | 1 |
| 2 | 2 | 1 |
| 3 | 3 | 1 |
| 4 | 4 | 1 |
| 5 | 5 | 1 |
| after `UNCACHE TABLE` | 5 | 0 |
The same mismatch is why `spark.catalog.isCached` returned `false` while the
optimizer was still
using the cache, and why `UNCACHE TABLE` was a no-op: both look the entry up
by plan.
This does **not** fix the other item described in #20104 — writes through
the DataFrame writer do
not invalidate the cache at all, because `HoodieSparkSqlWriter` only
refreshes the catalog table
when meta sync is enabled. That has a different mechanism and is better
handled separately, so
this PR deliberately does not close the issue.
### Summary and Changelog
- `HoodieFileIndex` gets an explicit `equals`/`hashCode` over `spark`,
`metaClient`, `schemaSpec`,
`options`, `includeLogFiles` and `shouldEmbedFileSlices`, excluding
`fileStatusCache`. The cache
an index happens to hold is a performance detail, not part of the table's
identity.
- New `TestHoodieFileIndexCaching` drives five cache/write cycles and
asserts the entry count
stays at one, with the equivalent Parquet cycle alongside it.
Considered and not taken: moving `fileStatusCache` into a second parameter
list, which Scala would
exclude from `equals` automatically. `HoodieFileIndex` is public API and
that would break
positional construction for downstream callers.
Not changed: `spark` is also identity-compared. It is stable within a
session so it does not cause
this, and excluding it would change cross-session semantics not examined
here.
No code was copied from elsewhere.
### Impact
No public API change and no change to query results. Plans for the same
table now compare equal
across a refresh, which is what lets Spark recache in place — the behaviour
a Parquet table
already has.
Worth stating because it looks alarming: if plans compare equal across a
write, can a lookup match
a stale entry? That is exactly how Parquet already behaves. Spark's contract
is that plans stay
equal across a refresh and `recacheByPlan` re-executes to replace the
contents; equality is table
identity, freshness is `refreshTable`'s job.
Blast radius: across the repository, 29 non-test files reference
`HoodieFileIndex` and none
compare instances — a search for `fileIndex ==`, `.equals(fileIndex)`,
`Set[HoodieFileIndex]` and
`Map[HoodieFileIndex, ...]` returns only the `unapply` extractors in the
three
`Spark*HoodiePruneFileSourcePartitions` rules, which match on type. No test
asserts on index
equality. The only consumer of this equality is Spark's own plan machinery.
### Risk Level
low
Verified by reverting `HoodieFileIndex.scala` to master and confirming the
new test fails at cycle
2 with `expected: <1> but was: <2>`, then passes with the change. Checkstyle
and scalastyle clean
on both modules.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
