[
https://issues.apache.org/jira/browse/SPARK-58932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58932:
-----------------------------------
Labels: pull-request-available (was: )
> TransformWithStateInPySparkStateServer leaks the accepted socket
> ----------------------------------------------------------------
>
> Key: SPARK-58932
> URL: https://issues.apache.org/jira/browse/SPARK-58932
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming
> Affects Versions: 4.0.0, 4.1.3
> Reporter: Jiwon Park
> Priority: Major
> Labels: pull-request-available
>
> TransformWithStateInPySparkStateServer.run() accepts a connection from the
> Python worker but never closes it. The task completion listener closes only
> the listening ServerSocketChannel.
> initStateServer() binds an ephemeral port per task, so every task leaks one
> socket. On a streaming query with a 10s trigger and 8 shuffle partitions this
> was measured at 45-51 CLOSE_WAIT sockets per minute, exhausting the ephemeral
> port range in about 10 hours. After that every new connection fails with
> BindException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
