Jiwon Park created SPARK-58932:
----------------------------------
Summary: TransformWithStateInPySparkStateServer leaks the accepted
socket
Key: SPARK-58932
URL: https://issues.apache.org/jira/browse/SPARK-58932
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark
Affects Versions: 4.0.0
Reporter: Jiwon Park
TransformWithStateInPySparkStateServer.run() accepts a connection from the
Python worker but never closes it. The task completion listener closes only the
listening ServerSocketChannel.
initStateServer() binds an ephemeral port per task, so every task leaks one
socket. On a streaming query with a 10s trigger and 8 shuffle partitions this
was measured at 45-51 CLOSE_WAIT sockets per minute, exhausting the ephemeral
port range in about 10 hours. After that every new connection fails with
BindException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
