Hello!

Please excuse me for asking this question, but I'm fighting a problem I don't 
understand:
I have written an application that forwards data from on TCPv4 stream to 
another, that is, you start the utility on hostA listening at the IP of hostA, 
portA, redirecting to hostB, portB. Now when hostC connects to hostA:portA vir 
portC, the traffic is forwarded to hostB:portB, and hostB's replies are 
forwarded back to hostC:portC.

That works well enough (I tested with Firefox on hostC and a webserver on 
hostB). I only have a problem: From time to time a shutdown(sock, 1) of the 
socket connected to hostB:portB seems to hand forever after hostC has closed or 
shutdown it socket.

The effect is that connections stay in CLOSE_WAIT like this:
# netstat -tn |grep 48123
tcp        1      0 172.20.17.252:64236     172.20.17.78:48123      CLOSE_WAIT
tcp        1      0 172.20.17.252:54347     172.20.17.78:48123      CLOSE_WAIT
tcp        1      0 172.20.17.252:51009     172.20.17.78:48123      CLOSE_WAIT
tcp        1      0 172.20.17.252:49899     172.20.17.78:48123      CLOSE_WAIT

Even though I set SO_LINGER option, the shutdown never completes, and I'll have 
to kill the process handling the connection.
I've read about SO_LINGER2, but being confused by the description I tried to 
read the source code. I don't understand (/usr/src/linux/net/ipv4/tcp.c, 
tcp_close()) the case when tp->linger2 >= 0: The value of tp->linger2 is not 
used anywhere; instead tcp_fin_time(sk) is used.

The doc (man 7 tcp) says:
       TCP_LINGER2 (since Linux 2.4)
              The  lifetime  of orphaned FIN_WAIT2 state sockets.  This option
              can be used to override the  system-wide  setting  in  the  file
              /proc/sys/net/ipv4/tcp_fin_timeout for this socket.  This is 
not(...)

Can somebody clarify?

Back to my application: I handle each direction (HostC->HostA->HostB (read), 
HostB->HostA->HostC (write)) in a separate thread, and I try to shutdown() the 
corresponding socket direction if I detect an EOF condition in any direction. 
typically read first. Occasionally I see an error "Transport
 endpoint is not connected" when shutting down the write connection on the 
socket connected to HostA. Is there a mistake on my side?

(D)[16049](172.20.16.36/13939): handler uses input socket 4
(D)[16049](172.20.16.36/13939): set SO_LINGER on socket 4 to 1, 0
(D)[16049](172.20.16.36/13939): connection_handler: use output socket 0
(D)[16049](172.20.16.36/13939): set SO_LINGER on socket 0 to 1, 0
(D)[16049](172.20.16.36/13939): connection_handler: connected to 
172.20.17.78/48123 in 99 microseconds
(D)[16049](172.20.16.36/13939): create input thread
(D)[16049](172.20.16.36/13939): create output thread
(D)[16049](input): received 147 bytes from 4
(D)[16049](input): sent 147 bytes to 0
(D)[16049](output): received 671 bytes from 0
(D)[16049](output): sent 671 bytes to 4
(D)[16049](input): copy 4 (172.20.16.36/13939) -> 0 (172.20.17.78/48123) 
finished with 0
(D)[16049](input): shutdown read direction (4 -> 0)
(D)[16049](input): shutdown socket 4 with 0
(D)[16049](input): shutdown socket 0 with 1
(D)[16049](172.20.16.36/13939): input thread 140526054442752 returned 0
(D)[16049](172.20.16.36/13939): wait for output thread 140526062835456
(D)[16049](output): copy 0 (172.20.17.78/48123) -> 4 (172.20.16.36/13939) 
finished with 0
(D)[16049](output): shutdown write direction (0 -> 4)
(D)[16049](output): shutdown socket 0 with 0
(N)[16049](output): failed to shutdown socket 0 with 0: Transport endpoint is 
not connected
(D)[16049](output): shutdown socket 4 with 1
(W)[16049](172.20.16.36/13939): output thread 140526062835456 returned -1
(D)[16049](172.20.16.36/13939): close inbound socket 4
(D)[16049](172.20.16.36/13939): close outbound socket 0
(D)[16049](172.20.16.36/13939): connection_handler finished
(D)[16049](172.20.16.36/13939): handler exiting after 17.038

Here are some code sketches (the whole thing is quite long and verbose) for 
both threads:

/* handle input data */
static  void    *input_thread(void *arg)
{
        handler_t       *handler = arg;
        char            ID[ID_MAX];
        tsap_t          *const i_conn = &handler->i_conn;
        tsap_t          *const o_conn = &handler->o_conn;
        thread_result_t result;
        int             i;

/*...*/
        result.i = socket_copy(i_conn->sock, o_conn->sock, &handler->i_bytes,
                               &handler->last_rd, ID);
        vlogf(LOG_DEBUG,
              "%s: copy %d (%s) -> %d (%s) finished with %d",
              ID, i_conn->sock, i_conn->r_addr.desc_real, o_conn->sock,
              o_conn->r_addr.desc_real, result.i);
        vlogf(LOG_DEBUG, "%s: shutdown read direction (%d -> %d)", ID,
              i_conn->sock, o_conn->sock);
        if ( do_shutdown(ID, i_conn->sock, SHUT_RD) != 0 )
                result.i = -1;
        if ( do_shutdown(ID, o_conn->sock, SHUT_WR) != 0 )
                result.i = -1;
        return(result.ptr);
}

/* handle output data */
static  void    *output_thread(void *arg)
{
        handler_t       *handler = arg;
        char            ID[ID_MAX];
        tsap_t          *const i_conn = &handler->i_conn;
        tsap_t          *const o_conn = &handler->o_conn;
        thread_result_t result;
        int             i;

        /*...*/
        result.i = socket_copy(o_conn->sock, i_conn->sock, &handler->o_bytes,
                               &handler->last_wr, ID);
        vlogf(LOG_DEBUG,
              "%s: copy %d (%s) -> %d (%s) finished with %d",
              ID, o_conn->sock, o_conn->r_addr.desc_real, i_conn->sock,
              i_conn->r_addr.desc_real, result.i);
        vlogf(LOG_DEBUG, "%s: shutdown write direction (%d -> %d)", ID,
              o_conn->sock, i_conn->sock);
        if ( do_shutdown(ID, o_conn->sock, SHUT_RD) != 0 )
                result.i = -1;
        if ( do_shutdown(ID, i_conn->sock, SHUT_WR) != 0 )
                result.i = -1;
        return(result.ptr);
}

The connection_handler process does this roughly (after having set up the 
sockets):
...
                if ( (err = pthread_create(&handler->input, NULL, input_thread,
                                           handler)) != 0 )
                {
                        result = -1;
                }
                vlogf(LOG_DEBUG, "%s: create output thread", handler->ID);
                if ( (err = pthread_create(&handler->output, NULL,
                                           output_thread, handler)) != 0 )
                {
                        result = -1;
                }
...
                if ( (err = pthread_join(handler->input, &ret.ptr)) == 0 )
                {
...
                }
        if ( (err = pthread_join(handler->output, &ret.ptr)) == 0 \
)
                {
...
        }
...
                if ( do_close(i_conn->sock) != 0 )
                        result = -1;
                if ( do_close(o_conn->sock) != 0 )
                        result = -1;

Regards,
Ulrich

Reply via email to