HeartSaVioR commented on code in PR #50600:
URL: https://github.com/apache/spark/pull/50600#discussion_r2049935483
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3583,11 +3583,15 @@ object SQLConf {
"than zero and less than INT_MAX.")
.createWithDefaultString("256MB")
- val ARROW_TRANSFORM_WITH_STATE_IN_PANDAS_MAX_RECORDS_PER_BATCH =
-
buildConf("spark.sql.execution.arrow.transformWithStateInPandas.maxRecordsPerBatch")
- .doc("When using TransformWithStateInPandas, limit the maximum number of
state records " +
- "that can be written to a single ArrowRecordBatch in memory.")
+ val ARROW_TRANSFORM_WITH_STATE_IN_PYSPARK_MAX_RECORDS_PER_BATCH =
+
buildConf("spark.sql.execution.arrow.transformWithStateInPySpark.maxRecordsPerBatch")
Review Comment:
Yeah we had a discussion about this and we seem to miss it. I'll address
altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
