viirya commented on code in PR #50301:
URL: https://github.com/apache/spark/pull/50301#discussion_r2029712711
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3391,6 +3391,19 @@ object SQLConf {
.intConf
.createWithDefault(10000)
+ val ARROW_EXECUTION_MAX_RECORDS_PER_OUTPUT_BATCH =
+ buildConf("spark.sql.execution.arrow.maxRecordsPerOutputBatch")
+ .doc("When using Apache Arrow, limit the maximum number of records that
can be output " +
+ "in a single ArrowRecordBatch to the downstream operator. If set to
zero or negative " +
+ "there is no limit. Note that the complete ArrowRecordBatch is
actually created but " +
+ "the number of records is limited when sending it to the downstream
operator. This is " +
+ "used to avoid large batches being sent to the downstream operator
including " +
+ "the columnar-based operator implemented by third-party libraries.")
+ .version("4.1.0")
+ .internal()
Review Comment:
It is internal config for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
