The way I look at it is that Apache HttpComponents is a pure Java library that implements HTTP semantics as completely and correctly as possible. The feature-richness is what distinguishes us from HttpURLConnection, and the simplicity and WORA-ness of pure Java (with a tiny dependency closure) is what distinguishes us from frameworks that make heavy or mandatory use of native libraries, such as Netty or the AWS Common Runtime. So I think we want to do what we can, while building directly on top of the JDK.
Newer JDKs bring new functionality that we can take advantage of. JDK8 gave us better async support with `CompletableFuture<T>`. JDK8 MR3 gave us native support for TLSv1.3 and ALPN, which is why we decided to skip JDK11. Looking ahead, JDK9 offers a standard API for Reactive Streams, JDK17 would give us native support for Unix-domain sockets (JEP 380), and JDK21 gives us virtual threads (albeit with temporary limitations around the use of `synchronized` that tend to result in thread pinning). Having said that, I can see three reasons to stay on JDK8: 1. JDK17+ is a huge lift for a foundational library like HttpComponents. A JDK17 or JDK21 minimum runtime requirement would prevent a lot of frameworks that use HttpComponents (such as my own) from upgrading, as we still have thousands and thousands of consumers running on JDK8, to say nothing of build-time complications that JDK17+ would introduce. I would imagine that Android developers would also have a ton of issues adopting a JDK17 library. 2. Many newer JDK features are usable indirectly, without us needing to upgrade HttpComponents itself. Virtual threads are a great example: libraries can compatibly adjust code (e.g. by migrating from `synchronized` to `ReentrantLock`) for forward-compatibility. Similarly, Unix domain sockets fit into existing JDK8 abstractions like `SocketAddress`, so we can already work with UDS types that are created by calling code, or by a small optional extension library. This kind of forward compatibility is something that the JDK maintainers put a lot of conscious effort into providing; see [1]. 3. I think the list of new features we get, even from JDK21, is still pretty short. We're not getting built-in support for QUIC or io_uring any time soon, for example; these facilities are largely the domain of C++/Rust programmers right now, and I think it's still going to take a long time for these features to filter into the JDK and/or OS networking stacks. Finally, a word about native libraries. The JDK maintainers have a long-term initiative towards what they call "platform integrity," which basically means that libraries cannot tamper with the internals of the JDK. Jigsaw and its strong encapsulation was only the start of this. In the future, similar restrictions are going to be placed on JNI [2]. Apache HttpComponents will not be able to load a native library -- an .so, .dylib, or .dll -- unless the application owner opts in by supplying a JVM flag. Given these trends, I think it's a good time to be a pure, well-behaved Java library, and I think we should continue to keep our distance from native libraries and not put something like quiche in our mandatory dependencies. So in conclusion, I suggest that we stay the course with JDK8. Some things we can work on include: 1. We should find backwards-compatible ways to expose new JDK features, such as Unix-domain sockets. Techniques we can use for this include optional extension libraries, multi-release jar files, and optional polyfill libraries for older JDKs (e.g. junixsocket [3] for Unix-domain sockets). This is similar to what we did with Conscrypt before JDK8 MR3 backported ALPN support to JDK8. 2. We should ensure that HTTP/1.1 works well with JDK21 virtual threads, and resolve any bottlenecks we discover (e.g. thread pinning). 3. We should experiment with virtual threads in the context of HTTP/2, since multiplexed protocols are a non-trivial problem with the blocking IO abstraction as Oleg mentioned. (I've never understood how this can be done without a multiplexed blocking feature in the language itself, such as the `select` statement in Go.) The JDK maintainers always appreciate useful feedback from library developers, and we may be able to help inform future work on virtual threads. [1] https://www.youtube.com/watch?v=2y5Pv4yN0b0&t=1577s [2] https://openjdk.org/jeps/8307341 On Mon, Mar 11, 2024 at 7:11 AM Oleg Kalnichevski <ol...@apache.org> wrote: > HttpClient 5.4 is going to be the most feature rich minor release > probably since 4.3. There are plenty of small and not so small features > and improvements in it. Overall it is going to be a great release. > > It will likely take a few more months to stablize it but somethime > around mid Summer we should be able to release the first GA. That is > the plan. > > However beyond 5.4 our development plans are unclear. We can settle for > no particular plan and simply react to feature requests from our users > or we could invest some efforts and time proactively. Both approaches > are perfectly valid given the scarcity of project resources. > > If we choose a more proactive stance there are several potential > features we could work on > > > core: > > * Better / more efficient implementation of channel (socket) timeout > handling in the async i/o reactor. What we currently have is quite bad > for a large number of concurrent sessions > > * CONNECT method support and connection tunneling for HTTP/2 > > > * HTTP/2 for the classic transport (now with introduction of virtual > threads (fibers) in Java 21 we could revisit the issue of the classic > i/o model not working well for multiplexing protocols and try to solve > the problem by utilizing two fibers per HTTP/2 data stream) > > > * JRE level upgrade > > > client: > > * Multipart entity support for the async transport > > * Transparent content compression / decompression for the async > transport > > * Query timeout / request deadline support > > * JRE level upgrade > > > The real 1'000'000 USD question for us as a project however what the > hell we are going to do about HTTP/3? > > HTTP/3 and QUIC dwarfs everytihng we have done so far in terms of > complexity. Do we have resources to pull it off? Do we attempt to build > our own QUIC layer or do we wait until QUIC is supported and provided > by JRE? Is QUIC even in scope for us? > > The subject of HTTP/3 support is so big that it probably deserves a > separate discussion. > > What do you, guys, think? How shall we proceed? > > Oleg > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org > For additional commands, e-mail: dev-h...@hc.apache.org > >
