voonhous commented on issue #19996:
URL: https://github.com/apache/hudi/issues/19996#issuecomment-5740750015
I compared `hudi.max-split-size` (Trino 483) with `hudi.target-split-size`
(`hudi-trino`). They split base files the same way:
- Trino 483: `HudiReadOptimizedDirectoryLister` sets each file's split size
to `min(fileLength, maxSplitSize)`, and `HudiSplitFactory` cuts chunks of that
size while more than 1.1x of it remains; the rest becomes the last split.
- `hudi-trino`: `HudiSplitFactory.createSplitsForBaseFile` runs the same
loop with the same `SPLIT_SLOP = 1.1`, using `targetSplitSize`.
For the same value, both produce the same splits. Neither is a strict
maximum: with the 10% slop a split can be up to 1.1x the setting.
| | `hudi.max-split-size` (Trino 483) | `hudi.target-split-size`
(`hudi-trino`) |
|---|---|---|
| Default | 120MB | 128MB |
| Session property | none | `target_split_size` |
| Validation | not null | at least 1B |
Only the default changes behavior. So the plan is to alias:
`@LegacyConfig("hudi.max-split-size")` on `setTargetSplitSize`. The open
decision is the default: unless `hudi-trino` keeps 120MB, catalogs that don't
set the property move from 120MB to 128MB splits, which should get a release
note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
