boneanxs opened a new pull request, #11652:
URL: https://github.com/apache/incubator-gluten/pull/11652
This PR fixes an OOM issue in transition error handling by making
`Transition.Factory.findTransition(..., otherwise)` lazy and by avoiding
full-plan stringification in `Transition.notFound`.
`otherwise` was passed by value (otherwise: Exception), so
`Transition.notFound(...)` was constructed eagerly at call sites.
`notFound` previously interpolated `$plan`, which triggers full
`SparkPlan.toString()` and can allocate huge strings for large plans, causing:
```
java.lang.OutOfMemoryError
at
java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at
java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.gluten.extension.columnar.transition.Transition$.notFound(Transition.scala:59)
at
org.apache.gluten.extension.columnar.transition.Transitions$.enforceReq(Transitions.scala:105)
at
org.apache.gluten.extension.columnar.transition.InsertTransitions.apply(Transitions.scala:33)
at
org.apache.gluten.extension.columnar.transition.Transitions$.insert(Transitions.scala:89)
at
org.apache.gluten.extension.GlutenColumnarRule.org$apache$gluten$extension$GlutenColumnarRule$$$anonfun$postColumnarTransitions$1(GlutenColumnarRule.scala:100)
at
org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85)
at
org.apache.gluten.extension.GlutenColumnarRule$$anonfun$postColumnarTransitions$2.apply(GlutenColumnarRule.scala:85)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
