atharvalade opened a new pull request, #3197:
URL: https://github.com/apache/iggy/pull/3197
## Which issue does this PR close?
Closes #3174
## Rationale
Sources (e.g. Postgres) wrap row data in a `DatabaseRecord` envelope while
sinks (e.g. Iceberg) expect flat JSON matching the target table schema — no
shared contract exists, producing silent null failures.
## What changed?
The Postgres source emits `{table_name, operation_type, timestamp, data:
{...}, old_data}` envelopes, but the Iceberg sink's Arrow JSON reader maps
these nested structures to top-level fields as null, silently violating
non-nullable constraints.
This adds a reusable `unwrap_envelope` transform to the connector SDK that
extracts a nested field (e.g. `data`) and promotes it as the top-level payload,
plus explicit envelope detection in the Iceberg sink that errors with an
actionable message instead of failing silently.
## Local Execution
- Passed
- Pre-commit hooks ran (fmt, clippy, license-headers, trailing-whitespace,
trailing-newline all clean; 119 tests pass across SDK + integration suites)
## AI Usage
1. Opus 4.6
2. used for codebase exploration and following existing transform patterns
3. All 8 new unit tests pass locally, clippy/fmt clean, existing 111 tests
unaffected
4. Yes, all code can be explained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
