hudi-agent commented on code in PR #19592:
URL: https://github.com/apache/hudi/pull/19592#discussion_r3948367315
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,31 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
Review Comment:
🤖 This bullet says a datasource write always sets the key, so clustering
triggered from one "takes the row-writer path unless you turn it off".
`ENABLE_ROW_WRITER` has a `withInferFunction` in `DataSourceOptions.scala` that
infers `false` when the operation is `bulk_insert`, meta fields are not
populated, and `hoodie.combine.before.insert` is `true` — and
`HoodieWriterUtils.setDefaultValue` honors the infer function, so the key lands
in the write config as `false`, not `true`. Clustering inheriting those
properties would then take the RDD path even though nobody set the config. It
might help to add a short caveat like "unless it is inferred to `false`
(bulk_insert without meta fields with combine-before-insert on)", so a reader
debugging why clustering did not use the row writer has the lead. @yihua could
you confirm this inference is the only case where the datasource write path
ends up with the key as `false`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/docs/clustering.md:
##########
@@ -202,6 +202,31 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely. That is
+ what `HoodieClusteringJob` (spark-submit or hudi-cli) and Hudi Streamer see,
since both build their write
+ config from raw properties. In-process async clustering from a Spark
datasource streaming write and
+ `CALL run_clustering` do not: both go through the datasource write defaults,
so they carry the key as `true`.
+ The fallback has not been stable across releases: it was `false` in 0.14.0,
0.14.2, 0.15.0 and 0.15.1, and
+ `true` in 0.14.1 and from 1.0.0 onwards. On this release it is `true`, so
those two paths use the row writer
+ by default as well.
+
+To force the RDD path, set `hoodie.datasource.write.row.writer.enable=false`
in the same properties the
Review Comment:
🤖 This section presents `hoodie.datasource.write.row.writer.enable` as if it
only selects the clustering rewrite path, but it is a general Spark write
config — its own documentation is "When set to true, will perform write
operations directly using the spark native `Row` representation", and it also
gates the row-writer path for bulk_insert ingestion and for Hudi Streamer
(`StreamSync.canUseRowWriter`). So setting it to `false` "in the same
properties the clustering job reads" will, for inline/async clustering attached
to a Streamer or datasource write, also disable the row writer for the
ingestion writes in that job. It would help to call that out here, and to note
that a standalone `HoodieClusteringJob` is the case where the setting affects
clustering only.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/versioned_docs/version-1.0.0/clustering.md:
##########
@@ -156,6 +156,37 @@ The available strategies are as follows:
consistent bucket index and only applicable to the Spark engine. Set
`hoodie.clustering.execution.strategy.class`
to
`org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy`.
+#### Row writer
+
+On Spark, the execution strategies above can rewrite the data either through
the row writer, which operates on
+a `Dataset<Row>` and avoids converting records to Avro, or through the older
RDD path. Which one runs is decided
+by a single config:
+
+| Config Name | Default | Description |
+|-------------|---------|-------------|
+| `hoodie.datasource.write.row.writer.enable` | `true` | When enabled,
clustering rewrites file groups through the Spark row writer instead of the RDD
path. This is the config's own default; the fallback applied when the config is
absent differs by release, see below.<br /><br />`Config Param:
ENABLE_ROW_WRITER`<br />`Since Version: 0.9.0` |
+
+Two things about that default are worth knowing, because they are not the same
statement:
+
+* The config itself defaults to `true`, and Spark datasource writes set it
explicitly, so clustering triggered
+ from a datasource write takes the row-writer path unless you turn it off.
+* Clustering also applies its own fallback when the config is **absent** from
the write config entirely. That is
+ what `HoodieClusteringJob` (spark-submit or hudi-cli) and Hudi Streamer see,
since both build their write
+ config from raw properties. In-process async clustering from a Spark
datasource streaming write and
+ `CALL run_clustering` do not: both go through the datasource write defaults,
so they carry the key as `true`.
+ The fallback has not been stable across releases: it was `false` in 0.14.0,
0.14.2, 0.15.0 and 0.15.1, and
+ `true` in 0.14.1 and from 1.0.0 onwards. On this release it is `true`, so
those two paths use the row writer
+ by default as well.
+
+To force the RDD path, set `hoodie.datasource.write.row.writer.enable=false`
in the same properties the
+clustering job reads.
+
+On 1.0.x there is one further condition: even with the config enabled, Hudi
falls back to the RDD path when the
Review Comment:
🤖 The 1.0.x condition is stated correctly (I checked
`HoodieDataTypeUtils.canUseRowWriter` at `release-1.0.0`), but the paragraph
omits that `parquet.avro.write-old-list-structure` defaults to **`true`**,
which means this fallback does not trigger unless the user has explicitly set
it to `false`. As written, a reader may assume the RDD fallback is a common
occurrence on 1.0.x. Adding "which defaults to `true`, so this only applies if
you have explicitly disabled it" would make the caveat actionable. Same
paragraph in the 1.0.1 and 1.0.2 copies.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
