Xiao-zhen-Liu opened a new pull request, #5966:
URL: https://github.com/apache/texera/pull/5966
### What changes were proposed in this PR?
Adds `CacheKeyUtil`, which computes a deterministic cache key for an output
port
from the part of the workflow that produces it. For a target output port,
the key
is built from:
- the target output port's identity;
- every operator upstream of it (the connected component feeding the port),
and
for each one: its physical and logical operator id, its execution init info
(which encodes the operator type and all of its parameters, including any
UDF
code), and the schema of each of its output ports;
- the edges among those operators, including the specific ports each edge
connects.
These are written to a JSON document with sorted keys and a sorted operator
and
edge order, so the same upstream computation always serializes to the same
bytes
regardless of map or set iteration order, and the document is hashed with
SHA-256
to produce the key. Because the key depends only on things that affect the
result,
any upstream change (a parameter edit, a rewiring, a schema change) produces
a
different key, while changes elsewhere in the workflow do not.
Matching is collision-safe. `sameComputation` compares the hash first and,
on a
hash match, also compares the full JSON, so the extremely unlikely case of
two
different computations hashing to the same value is reported as a non-match
rather
than reusing a result the port was not computed from.
This is the first piece of the operator output port result cache.
`CacheKeyUtil`
is a pure function with no database or storage dependency, and nothing calls
it
yet, so it changes no existing behavior. The cache table (companion PR) and
the
service that uses both land separately.
### Any related issues, documentation, discussions?
Part of #5882 (umbrella #5881). Design discussion: #5880.
### How was this PR tested?
New unit test `CacheKeyUtilSpec` (8 cases): the key is stable across repeated
computation, changes when an operator's configuration changes, differs for
different ports, is stable for a source operator with no upstream, and
includes
only upstream operators (not downstream); and `sameComputation` matches equal
keys, rejects a fabricated hash collision by comparing the full JSON, and
rejects
keys with different hashes. `sbt "WorkflowExecutionService/testOnly
*CacheKeyUtilSpec"` — all 8 pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (Claude Code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
