Re: Authorization Header Limitations and Executor Behavior Clarification

[Daniel Fuchs]

Hi,

Thank you for trying the new HTTP client!

I believe both issues described here might have been fixed.
At least JDK 11 should let you specify your own authorization
header, and I have verified that a simple GET request works
with an inline executor.

However there could be some other issues explaining the
behavior you have observed - please continue reading:

1. We do have some tests to verify that it is possible to
   supply custom authorization headers to implement e.g. a
   DIGEST authentication on top of the API. These tests
   are included on our test base and run on regular basis.

   An important point to note is that the
   java.io.IOException: Invalid auth header
   is thrown when the server respond with an Unauthorized code
   (401/407) but does not include the expected
   WWW-Authenticate or Proxy-Authenticate headers.
   It would be helpful to know which headers the server is
   sending back.

   For troubleshooting it's possible to enable logging of
   headers on the java command line with
      -Djdk.httpclient.HttpClient.log=headers

   I realize the 'Invalid auth header' is not a very helpful
   message and I will log a JBS issue to examine if we can
   do better (maybe we should not throw an exception but
   just log it).

2. I have run a simple test to verify that you could supply an
   inline executor. It worked well for GET requests.

   Note however that in that case, you must be very careful of
   what code you trigger in dependent actions you might register
   with the returned CompletableFuture, as those will also execute
   in your inline executor and might therefore wedge the selector
   thread.

If any of these issue still shows up - then please could you try
the same with the java.net.http module in JDK 11?
Then reply to this mail with some more information - in particular
WRT to the header values you're trying to set, and what the server
sends back, and a thread dump to show where the inline executor
is wedged if the problem is still there, together with the
line of code that calls HttpClient::send/sendAsync as it
would be useful to know which BodyHandler was used.

best regards,

-- daniel


On 25/05/2018 19:03, danthonywalker wrote:

On Java 10.0.1 (build 10.0.1+10), the incubating HttpClient has 2 behaviors that I believe are problematic and should either be fixed or documented to better highlight their limitations.

The first concerns customized Authorization headers. From what I understand, Authorization headers outside of "Basic" are disallowed by the API. For example, when the type of Authorization is set to "Bot" or "Bearer", the following exception is thrown

    Caused by: java.util.concurrent.CompletionException:
    java.io.IOException: Invalid auth header
    at
    
java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
    at
    
java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
    at
    
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1074)
    at
    
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at
    
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
    at
    
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649)
    at
    
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
    at
    
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at
    
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)
    Caused by: java.io.IOException: Invalid auth header
    at
    
jdk.incubator.httpclient/jdk.incubator.http.AuthenticationFilter.lambda$response$1(AuthenticationFilter.java:211)
    at java.base/java.util.Optional.orElseThrow(Optional.java:397)
    at
    
jdk.incubator.httpclient/jdk.incubator.http.AuthenticationFilter.response(AuthenticationFilter.java:210)
    at
    
jdk.incubator.httpclient/jdk.incubator.http.MultiExchange.responseFilters(MultiExchange.java:195)
    at
    
jdk.incubator.httpclient/jdk.incubator.http.MultiExchange.lambda$responseAsyncImpl$7(MultiExchange.java:273)
    at
    
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    ... 7 more

Adding the following hack fixes the issue regarding using a "Bot" type (code is in Kotlin). The solution was taken from here: https://clevertap.com/blog/java-9-httpclient-does-not-allow-custom-authorization-headers/

    httpRequest.javaClass.getDeclaredMethod("setSystemHeader",
    String::class.java, String::class.java).apply {
         isAccessible = true // Allows access to an internal method to
    add Authorization header for httpRequest
         invoke(httpRequest, "authorization",
    requestHeaders["authorization"])
    }


However, "Bearer" still fails with the same exception.

Either this restriction should be lifted (I see no reason to restrict it), or add this behavior to the Javadocs.

The second issue regards setting an Executor when building a HttpClient. There is problematic behavior regarding it when using an Executor under 1 of 2 scenarios.

When an Executor uses the current running thread the request never completes and blocks the current thread. I have not done any in-depth debugging to pinpoint the exact cause. The behavior I expect from this is the thread is blocked as it waits for the request to come through, but it should come through eventually; as if it was a sync API.

When an Executor only uses a single thread (more specifically, if I use Executors.newSingleThreadExecutor), the same behavior occurs; except the behavior in this case should clearly be that the work is offloaded to another thread, allowing the application to continue, but it doesn't, it still gets stuck at the request call.

I have received an exception before when running on the current thread that relates to some method being invoked on the same thread as another method. However, when retrying these scenarios for this posting, I was unable to replicate this behavior, and thus both Executor scenarios produce the same behavior.

Either the Executor behavior should be fixed or its limitations should be documented.

Sorry if this post isn't following a specified format typical for mailing listings, I have never participated in one before, but I hope these concerns can be considered.