DanielLeens commented on PR #12846: URL: https://github.com/apache/seatunnel/pull/12846#issuecomment-6258266706
Thanks @wgzhao for running the exact `Dataset.create` call against 0.33.0, 0.39.0 and 1.0.0-beta.2 before changing the design. That evidence settles it: the incompatibility is in Lance's native type system, so a dependency bump will not unblock this PR. My preference is option 1. - Option 2 (`List<Struct<key, value>>`) persists a different Arrow type from the one the connector claims to write. Once users have datasets in that shape, switching to a native Map later would be a storage-format change. It would also be write-only for now, because the source side has no nested-type read support (the `struct|list|map` TODO in `LanceTypeMapper`). A sink that writes data the paired source cannot read back is a surprising contract to ship. - Option 1 keeps the connector honest. A clear "Map is not supported" failure at schema-build time, plus a corrected type-mapping table in `docs/en/connectors/sink/Lance.md` and `docs/zh/connectors/sink/Lance.md`, is a small and safe change. Once a published Lance runtime accepts Arrow Map, the canonical `Map(entries: Struct<key, value>)` work from this PR can be revisited. If you go with option 1, I suggest reusing this PR (retitling it to describe the fail-fast behavior) or opening the smaller PR you described. Please keep a unit test that asserts the exact error for a Map column, and remove the Map case from the Lance E2E so the mandatory Build can go green. Committers may have a different view on the storage-contract question, so please treat this as my recommendation rather than a final decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
