HeartSaVioR opened a new pull request, #50110: URL: https://github.com/apache/spark/pull/50110
### What changes were proposed in this pull request? This PR proposes to fix the logic of serializer in TWS PySpark version to NOT materialize the output entirely. This PR changes the logic of creating a list to create a generator instead, so that it can be lazily consumed. ### Why are the changes needed? Without this PR, all the outputs are materialized when JVM signals to Python worker that there is no further input (at task completion), which brings up two critical issues: * downstream operator can only see outputs after TWS operator processes all inputs * all the outputs are materialized into "memory" in Python worker, which could lead memory issue ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing UTs. I've confirmed manually below: * Before this PR, all the outputs are available after processing all inputs * After this PR, outputs are available during processing inputs ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
