atharvalade opened a new issue, #3174:
URL: https://github.com/apache/iggy/issues/3174
The Postgres source wraps all row data in a `DatabaseRecord` envelope:
```json
{
"table_name": "tpch.lineitem",
"operation_type": "SELECT",
"timestamp": "2026-04-24T19:57:00Z",
"data": {"id": 1, "l_orderkey": 123, ...},
"old_data": null
}
```
The Iceberg sink's Arrow JSON reader expects flat JSON matching the target
table schema:
```json
{"id": 1, "l_orderkey": 123, ...}
```
These two formats are fundamentally incompatible. There is no shared message
schema spec or envelope definition between sources and sinks. Connecting the
Postgres source to the Iceberg sink without the (newly added) `flat_json_output
= true` flag produces silent failures because Arrow maps the nested `data`
object to the top-level `id` field as null, then fails on non-nullable
constraints.
**Fix**: Either define a standard connector message envelope that all sinks
know how to unwrap, or have the sink explicitly handle the `DatabaseRecord`
format (extract the `data` field before Arrow conversion). The current
workaround (`flat_json_output`) is a band-aid that puts the burden on the user
to know about the incompatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
