goutamadwant opened a new issue, #13174:
URL: https://github.com/apache/seatunnel/issues/13174
### Search before asking
- [x] I searched existing issues and found no matching projection-stability
bug report.
### What happened
Zeta SQL `VECTOR_REDUCE` with `RANDOM_PROJECTION` or
`SPARSE_RANDOM_PROJECTION` creates a different projection matrix on successive
calls. Identical vectors can therefore produce different reduced vectors, and
different rows do not share a consistent projection. This makes similarity
comparisons between reduced embeddings unreliable and makes results depend on
evaluation order and concurrent calls.
The implementation keeps one static `Random(42)` and consumes its evolving
state each time either projection matrix is constructed. A fixed initial seed
does not make successive calls use the same matrix.
Expected: for the same method and source/target dimensions, every input row
uses the same projection, independently of call order, threads, transform
instances and process restart.
This is a newly reproduced correctness bug in the feature delivered for
#9782, not an incomplete implementation of that feature. #9782 should remain
closed.
### SeaTunnel Version
Reproduced against `dev` at `f54bb5876bd282b88181b6d93c9f59ec5dd38621`
(`3.1.0-SNAPSHOT`). Released versions were not separately tested.
### SeaTunnel Config
The regression extends the existing vector SQL E2E with these expressions
for an input vector `[1.0, 2.0, 3.0, 4.0, 5.0]`:
```sql
SELECT
L2_DISTANCE(
VECTOR_REDUCE(vector_field, 3, 'RANDOM_PROJECTION'),
VECTOR_REDUCE(vector_field, 3, 'RANDOM_PROJECTION')
) AS gaussian_self_distance,
L2_DISTANCE(
VECTOR_REDUCE(vector_field, 3, 'SPARSE_RANDOM_PROJECTION'),
VECTOR_REDUCE(vector_field, 3, 'SPARSE_RANDOM_PROJECTION')
) AS sparse_self_distance
FROM dual
```
Both distances should be zero. The complete tested job configuration is
[func_vector.conf at the fix
commit](https://github.com/goutamadwant/seatunnel/blob/f0b95132b959943414b7033b6e0274133a74ddfb/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/sql_transform/func_vector.conf).
### Running Command
With the transform regression tests and E2E prerequisites built:
```shell
./mvnw -pl seatunnel-transforms-v2
'-Dtest=VectorFunctionTest#testProjection*,SQLVectorFunctionTest#testProjectionStableAcrossRowsAndTransforms'
test
RUN_ENGINES=zeta ./mvnw -f
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/pom.xml
-DskipUT=true -DskipIT=false '-Dit.test=TestSQLIT#testVectorFunctions' verify
```
Before the fix, 9 of the 10 new unit/SQL regression cases failed. For the
Docker negative control, only `VectorFunction.class` was replaced with the
class compiled from the original base source; the rest of the runtime and job
configuration were unchanged. The assertion sink rejected nonzero
self-distances. Restoring the patched artifact made the same Docker test pass.
### Error Exception
The Docker negative control produced Gaussian self-distance
`0.6619507881116014` and sparse self-distance `1.9400093716986018` for the
identical input vector. The Gaussian assertion reported:
```text
AssertConnectorException: ErrorCode:[ASSERT-01], ErrorDescription:[Rule
validate failed]
fieldName=gaussian_self_distance, fieldType=DOUBLE, equalTo=0
```
This is a correctness failure; an ordinary job without the assertion sink
may complete while producing inconsistent projections.
### Zeta or Flink or Spark Version
Zeta from the same `3.1.0-SNAPSHOT` revision. Calcite is outside this bug's
scope and its implementation is unchanged.
### Java or Scala Version
Java 11 and 17 unit/SQL validation; Docker-backed Zeta runtime on Java 11.
### Screenshots
Not applicable; the failure is captured by deterministic tests and
assertion-sink output.
### Proposed fix and validation
Create a locally seeded generator for each projection-matrix construction,
preserving the existing coefficient generation and sparse scaling. No global
cache, SQL syntax, option, schema or checkpoint-format change is needed.
Previously stored projected vectors must be rebuilt from the original
embeddings before comparing or mixing them with new results.
The fix is available on
[fix/stable-zeta-vector-projection](https://github.com/apache/seatunnel/compare/dev...goutamadwant:fix/stable-zeta-vector-projection?expand=1).
All 1,967 transform tests passed on each of Java 11 and 17, and the Docker
Zeta regression passed. Hosted CI is separate from these local results. Actual
worker-crash recovery and other engine matrices were not run; process-restart
determinism is covered by fresh-JVM tests.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]