Yeh monitor is present but for some cases like long running job I found App master is idle.so it will end up closing the App master’s channel so job will not be completed.
So needed a mechanism to close only invalid connections . On Wed, 28 Feb 2018 at 10:54 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Spark already has code to monitor idle connections and close them. > That's in TransportChannelHandler.java. > > If there's anything to do here, it's to allow all users of the > transport library to support the "close idle connections" feature of > that class. > > On Wed, Feb 28, 2018 at 9:07 AM, sandeep_katta > <sandeep0102.opensou...@gmail.com> wrote: > > In case of client mode App Master and Driver are in different JVM > process,the > > port opened by the Driver is vulnerable for flooding attacks as it is not > > closing the IDLE connections. > > > > I am thinking to fix this issue using below mechanism > > > > 1.Expose configuration to close the IDLE connections as secondary > developers > > can decide based on their use case > > > > 2.Maintain the SET of validConnections,the condition for > validConnections is > > *TransportChannelHandler.channelRead* event will be fired for the valid > > connections. > > > > 3.Whenever there is timeout(no communication happened across the channel > for > > x time),close the connection. > > > > if !validConnectionSet then > > close the connection > > > > Note: Timeout value is configurable using the config > "spark.network.timeout" > > > > > > can this solution be used to solve this issue,if any suggestions are also > > welcomed. > > > > Regards > > Sandeep > > > > > > > > -- > > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > > > -- > Marcelo >
