Discussion on root cause analysis of JDK-7052625 : com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

[michael cui]

Hi,

I would like to discuss my current root cause analysis of JDK-7052625  : 
com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently


As JDK-6725892  
stated, the purpose of this regression test is testing bad http 
connections can be handled correctly which including

+ send no request
+ send an incomplete request
+ fail to read the response completely.

test3() method will start 20 threads for each type listed above at same 
time. So totally 60 threads started in test3(). Each thread will open 
connection to httpserver and simulate the normal or bad http request to 
see if http server can handle them correctly. (20 threads for incomplete 
read, 20 threads for incomplete write, 20 threads for read/write normal 
case)


Those threads will be started at same time. Among them, 40 threads using 
sleep to simulate bad request.


The http server created by the following api call :
s1 = HttpServer.create (addr, 0);

According API doc 
 
and ServerSocket.java source code, the second parameter is backlog of 
socket which is the maximum number of queued incoming connections to 
allow on the listening socket. Queued TCP connections exceeding this 
limit may be rejected by the TCP implementation.. The default value 50 
will be used if it was set to zero (See api doc 
 
and ServerSocket.java )).


Since in test3(), 40 threads out of total 60 threads will simulate bad 
http request by sleeping either at reading or writing, there could be a 
very little possibility that httpserver 's socket connection queue reach 
his limit (50 for default value) and some tcp connection will be rest at 
that situation.


This could be the root cause of this intermittently failure.

Test result of the original version :
0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
6 failure on windows for 1 runs
28 failures on mac for 1 runs

By increasing the thread number of bad request, we can observe that the 
frequency of failure will be increased.


Test result of fix version in which backlog of httpserver was changed 
from 0 to 100.

0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
0 failure on windows for 1 runs
0 failures on mac for 1 runs

It seems to me that using default 0 for backlog of httpserver could be 
root cause of this intermittently failure.
Are we comfortable with this analysis? If it is the root cause, could 
setting backlog as 100 be a suggest fix?


Thanks,
Michael Cui

Re: Discussion on root cause analysis of JDK-7052625 : com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

[David Holmes]

Sounds like a question for net-dev more than core-libs - cc'd.

David

On 20/02/2014 4:11 PM, michael cui wrote:

On 02/18/2014 12:51 AM, michael cui wrote:

Hi,

I would like to discuss my current root cause analysis of JDK-7052625
: com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

As JDK-6725892 
stated, the purpose of this regression test is testing bad http
connections can be handled correctly which including
+ send no request
+ send an incomplete request
+ fail to read the response completely.

test3() method will start 20 threads for each type listed above at
same time. So totally 60 threads started in test3(). Each thread will
open connection to httpserver and simulate the normal or bad http
request to see if http server can handle them correctly. (20 threads
for incomplete read, 20 threads for incomplete write, 20 threads for
read/write normal case)

Those threads will be started at same time. Among them, 40 threads
using sleep to simulate bad request.

The http server created by the following api call :
s1 = HttpServer.create (addr, 0);

According API doc

and ServerSocket.java source code, the second parameter is backlog of
socket which is the maximum number of queued incoming connections to
allow on the listening socket. Queued TCP connections exceeding this
limit may be rejected by the TCP implementation.. The default value 50
will be used if it was set to zero (See api doc

and ServerSocket.java )).

Since in test3(), 40 threads out of total 60 threads will simulate bad
http request by sleeping either at reading or writing, there could be
a very little possibility that httpserver 's socket connection queue
reach his limit (50 for default value) and some tcp connection will be
rest at that situation.

This could be the root cause of this intermittently failure.

Test result of the original version :
0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
6 failure on windows for 1 runs
28 failures on mac for 1 runs

By increasing the thread number of bad request, we can observe that
the frequency of failure will be increased.

Test result of fix version in which backlog of httpserver was changed
from 0 to 100.
0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
0 failure on windows for 1 runs
0 failures on mac for 1 runs

It seems to me that using default 0 for backlog of httpserver could be
root cause of this intermittently failure.
Are we comfortable with this analysis? If it is the root cause, could
setting backlog as 100 be a suggest fix?

Thanks,
Michael Cui


Could anyone provide some insight on this analysis?

Michael Cui

Re: Discussion on root cause analysis of JDK-7052625 : com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

[Chris Hegarty]

Michael,

I’m ok with your analysis, and suggested fix. 

From the original test output, in the bug description, I can see that there are 
58 println’s with "Request from:" for test3, and two "Worker: Error writing to 
server”. This would tend to support your analysis that that server, in some 
cases, is not accepting the barrage of requests.

Please provide a webrev/changeset and I will sponsor the change for you.

-Chris.

On 20 Feb 2014, at 08:25, michael cui  wrote:

> Hi,
> 
> I would like to discuss my current root cause analysis of JDK-7052625  : 
> com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently
> 
> As JDK-6725892  stated, the 
> purpose of this regression test is testing bad http connections can be 
> handled correctly which including
> + send no request
> + send an incomplete request
> + fail to read the response completely.
> 
> test3() method will start 20 threads for each type listed above at same time. 
> So totally 60 threads started in test3(). Each thread will open connection to 
> httpserver and simulate the normal or bad http request to see if http server 
> can handle them correctly. (20 threads for incomplete read, 20 threads for 
> incomplete write, 20 threads for read/write normal case)
> 
> Those threads will be started at same time. Among them, 40 threads using 
> sleep to simulate bad request.
> 
> The http server created by the following api call :
> s1 = HttpServer.create (addr, 0);
> 
> According API doc 
> 
>  and ServerSocket.java source code, the second parameter is backlog of socket 
> which is the maximum number of queued incoming connections to allow on the 
> listening socket. Queued TCP connections exceeding this limit may be rejected 
> by the TCP implementation.. The default value 50 will be used if it was set 
> to zero (See api doc 
> 
>  and ServerSocket.java )).
> 
> Since in test3(), 40 threads out of total 60 threads will simulate bad http 
> request by sleeping either at reading or writing, there could be a very 
> little possibility that httpserver 's socket connection queue reach his limit 
> (50 for default value) and some tcp connection will be rest at that situation.
> 
> This could be the root cause of this intermittently failure.
> 
> Test result of the original version :
> 0 failure on Linux for 1 runs.
> 0 failure on solaris for 1 runs.
> 6 failure on windows for 1 runs
> 28 failures on mac for 1 runs
> 
> By increasing the thread number of bad request, we can observe that the 
> frequency of failure will be increased.
> 
> Test result of fix version in which backlog of httpserver was changed from 0 
> to 100.
> 0 failure on Linux for 1 runs.
> 0 failure on solaris for 1 runs.
> 0 failure on windows for 1 runs
> 0 failures on mac for 1 runs
> 
> It seems to me that using default 0 for backlog of httpserver could be root 
> cause of this intermittently failure.
> Are we comfortable with this analysis? If it is the root cause, could setting 
> backlog as 100 be a suggest fix?
> 
> Thanks,
> Michael Cui

RFR for JDK-7052625 : com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

[michael cui]

On 02/20/2014 07:24 PM, Chris Hegarty wrote:

Michael,

I’m ok with your analysis, and suggested fix.

 From the original test output, in the bug description, I can see that there are 58 println’s 
with "Request from:" for test3, and two "Worker: Error writing to server”. This 
would tend to support your analysis that that server, in some cases, is not accepting the 
barrage of requests.

Please provide a webrev/changeset and I will sponsor the change for you.

Thank you very much on the review and sponsor!

webrev link : 
http://cr.openjdk.java.net/~tyan/michael/JDK-7052625/webrev.00/ 





-Chris.

On 20 Feb 2014, at 08:25, michael cui  wrote:


Hi,

I would like to discuss my current root cause analysis of JDK-7052625  : 
com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

As JDK-6725892  stated, the 
purpose of this regression test is testing bad http connections can be handled 
correctly which including
+ send no request
+ send an incomplete request
+ fail to read the response completely.

test3() method will start 20 threads for each type listed above at same time. 
So totally 60 threads started in test3(). Each thread will open connection to 
httpserver and simulate the normal or bad http request to see if http server 
can handle them correctly. (20 threads for incomplete read, 20 threads for 
incomplete write, 20 threads for read/write normal case)

Those threads will be started at same time. Among them, 40 threads using sleep 
to simulate bad request.

The http server created by the following api call :
s1 = HttpServer.create (addr, 0);

According API doc 

 and ServerSocket.java source code, the second parameter is backlog of socket which is the 
maximum number of queued incoming connections to allow on the listening socket. Queued TCP 
connections exceeding this limit may be rejected by the TCP implementation.. The default 
value 50 will be used if it was set to zero (See api doc 

 and ServerSocket.java )).

Since in test3(), 40 threads out of total 60 threads will simulate bad http 
request by sleeping either at reading or writing, there could be a very little 
possibility that httpserver 's socket connection queue reach his limit (50 for 
default value) and some tcp connection will be rest at that situation.

This could be the root cause of this intermittently failure.

Test result of the original version :
0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
6 failure on windows for 1 runs
28 failures on mac for 1 runs

By increasing the thread number of bad request, we can observe that the 
frequency of failure will be increased.

Test result of fix version in which backlog of httpserver was changed from 0 to 
100.
0 failure on Linux for 1 runs.
0 failure on solaris for 1 runs.
0 failure on windows for 1 runs
0 failures on mac for 1 runs

It seems to me that using default 0 for backlog of httpserver could be root 
cause of this intermittently failure.
Are we comfortable with this analysis? If it is the root cause, could setting 
backlog as 100 be a suggest fix?

Thanks,
Michael Cui

Michael Cui

Re: RFR for JDK-7052625 : com/sun/net/httpserver/bugs/6725892/Test.java fails intermittently

[Chris Hegarty]

On 20 Feb 2014, at 12:53, michael cui  wrote:

> On 02/20/2014 07:24 PM, Chris Hegarty wrote:
>> Michael,
>> 
>> I’m ok with your analysis, and suggested fix. 
>> 
>> From the original test output, in the bug description, I can see that there 
>> are 58 println’s with "Request from:" for test3, and two "Worker: Error 
>> writing to server”. This would tend to support your analysis that that 
>> server, in some cases, is not accepting the barrage of requests.
>> 
>> Please provide a webrev/changeset and I will sponsor the change for you.
>> 
> Thank you very much on the review and sponsor!
> 
> webrev link : http://cr.openjdk.java.net/~tyan/michael/JDK-7052625/webrev.00/

Pushed. http://hg.openjdk.java.net/jdk9/dev/jdk/rev/e5a09bc70308

-Chris.

Re: Add IDN support to existing net APIs

[Michael McMahon]

On 20/02/14 03:15, Jonathan Lu wrote:


Hi Michael,

Thanks a lot for your comments.

On Wed, Feb 19, 2014 at 6:28 PM, Michael McMahon 
mailto:michael.x.mcma...@oracle.com>> 
wrote:


I think it's a good idea. Before changing anything though,
we would need to:

 1. identify all APIs that could potentially be affected and
figure out what is
needed for each. For instance:
 1. InetAddress
 2. SocketPermission
 3. URLPermission
 4. HttpURLConnection
 5. URL/URI
 6. ...

Right, I assume all  APIs which accept string-represented host names 
would be affected.

It that may not be a big change from point of view.

1.


 1. understand how it relates to URIs and IRIs. I haven't looked
into this much
but it seems that there might be potential for confusion given
that these entities
have their own encoding schemes already. I'm sure the issues
are all well thrashed out
already, but I'd like to see exactly how it will work in Java
before we start the work

For the potential confusion, were you meaning a confusion between URI 
encoding (RFC-2396 [1]) and IDN ?


Right. As I said, it may just be a matter of checking RFC2396 (or its 
successor RFC 3986).

I think RFC 3986 references IDNs.

Michael


Maybe, the work is within the scope of a small JEP, if nothing
else to define the scope
of the work.

Michael.


Copying core-libs-dev and security-dev to get more comments.

Many thanks

- Jonathan

[1] http://www.ietf.org/rfc/rfc2396.txt



On 19/02/14 04:00, Jonathan Lu wrote:

Hi net-dev,

If a Java application tries to support International Domain Names
(IDN) [1], which was firstly brought in Java6,
 it has to write additional code or wrap java.net.IDN API [2] to
make the conversion each time it tries to resolve
a non-ASCII domain name, and use the converted punycode to make
connections to remote host, like:

java.net.IDN.toASCII("电 脑.info ");

It is easier for newly writen applications, since a wrappers can
be made in early design stage, but if
it comes to existing Java applications and third-party libraries,
IDN support would involve much more effort
like modifying the existing code base, inspecting libraries
interfaces/implementations, or even
re-implement some of the functions.

I'm wondering if it is possible to add IDN support into existing
Java APIs, like
java.net.InetAddress.getByName(), if JDK itself supports IDN, any
existing applications or libraries would
benefit from supporting IDN automatically without any source code
modifications.

Of course there's security risks regarding IDN homograph attacks
[1][2], so it may not be a
good idea to change the default behavior right now. But it would
still work if a simple switch can be
added into current JDK.
For example, by defining following the property in command like
options like

-Djava.net.AutoIDN=true

Any existing Java applications who wishes to support IDN feature
will be able to support IDN
without any changes to its source code or re-compilation.

What's your opinion on this ? any comment is welcome.

Thank you

- Jonathan Lu

---

[1] http://en.wikipedia.org/wiki/Internationalized_domain_name
[2] http://download.java.net/jdk8/docs/api/java/net/IDN.html
[3] http://www.unicode.org/reports/tr36/
[4] http://en.wikipedia.org/wiki/IDN_homograph_attack

Re: RFR: JDK-8015692 - java.net.BindException is thrown on Windows XP when HTTP server is started and stopped in the loop.

[Chris Hegarty]

Mark,

I agree with you, there is certainly some additional co-ordination needed 
between the thread invoking the stop method and the dispatcher thread.

I wonder why you needed to add the selectNow() and the close() after you have 
joined the dispatcher thread? Since you are guaranteed that the dispatcher 
thread will have exited before join() returns?

-Chris.

On 17 Feb 2014, at 01:20, Mark Sheppard  wrote:

> Hi
> 
>Please oblige and review the changes in the webrev
> 
> http://cr.openjdk.java.net/~msheppar/8015692/jdk9/webrev/
> 
> to address the issue raised in the bug
> 
> https://bugs.openjdk.java.net/browse/JDK-8015692
> 
> Summary:
> a series Junit tests which start stop instances of an 
> com.sun.net.httpserver.HttpServer failed due to
> java.net.BindException: Address already in use: bind
> 
> This was raised against Windows XP, but the sample test to reproduce the issue
> was run on Windows 7, and the problem was seen to occur on this OS also.
> The sample was run against jdk7, jdk8 and jdk9: reproducible on each.
> 
> On investigation it  appears that some additional co-ordination is required 
> between the
> HttpServer's  (actually SereverImpl) dispatcher thread and the thread 
> invoking the stop
> method. This change has amended the stop method to wait for the Dispatcher 
> thread to complete, then
> invokes the selector's selectNow, to handled cancelled events, and closes the 
> selector.
> The selector.close() has been removed from the Dispatcher's run method.
> 
> regards
> Mark