wgzhao commented on PR #12846:
URL: https://github.com/apache/seatunnel/pull/12846#issuecomment-6250492381
Thanks for the review. I checked the runtime compatibility before changing
the design any
further, and the incompatibility is in Lance's native type system rather
than in the schema
this PR constructs.
I ran the exact `Dataset.create` call with a canonical Arrow Map schema
against three Lance
Java SDK versions on macOS arm64:
| Lance Java SDK | Result |
| --- | --- |
| 0.33.0 (current) | `LanceError(Schema): Unsupported data type: Map(...)` |
| 0.39.0 (latest stable) | same error |
| 1.0.0-beta.2 (latest published) | same error |
So upgrading `lance-core` does not make Map work; no published Lance runtime
accepts the
Arrow Map type yet.
I also checked an alternative representation: Lance does accept
`List<Struct<key, value>>` (`Dataset.create` succeeds with 0.39.0). It is
not the same
persisted type as an Arrow Map, and the Lance source in this connector still
has no nested
type read support (`LanceTypeMapper` has a `struct|list|map` TODO), so it
would provide write
support only.
Could you confirm which direction you prefer?
1. Defer or close this PR until Lance supports Arrow Map natively. I can
follow up with a
smaller PR that fails fast with a clear "Map is not supported" error and
corrects the
documented type mapping.
2. Represent SeaTunnel `MAP` as `List<Struct<key, value>>` in the Lance sink
and document the
storage contract.
I am happy to implement whichever option you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
