During a maven publish if the server closes the TCP connection after the upload of an asset sometimes the publish will fail with a failed to respond while attempting to upload the checksum. The correct behavior to wanting to send more HTTP requests on a closed TCP connection, is to open a new TCP connection. If the client and server have a low latency connection, say in the same data center, when the connection is closed a new one appears to reliably be opened. During higher latency scenarios, in our case when the client and server are on different continents, maven attempts to reuse the connection that the server has already closed receiving a failed to respond error about half the time. On even slower connections we get a Connection or outbound has closed error.
Unfortunately, this connection does not get retried. It immediately fails to build no matter how many retries are configured. I believe this is the known https://issues.apache.org/jira/browse/MDEPLOY-162 If these requests were retried on this error, I suspect the entire situation would not have been a problem. The lack of retries go through different code paths depending on the transport implementation. The wagon transport fails with org.apache.maven.wagon.providers.http.httpclient.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity. Fixing the wagon transport requires changes in maven-wagon and in maven-resolver. In particular the WagonHttpEntity needs to use a stream supplier in order to be retryable. My colleague sketched out these changes a year ago. See https://github.com/mmarston/maven-wagon/commit/4419140ff7794a548f3f9cb9e320fcd20420b5e0 and https://github.com/mmarston/maven-resolver/commit/cf00080ac546bda588bd86a44f93c037f7638145 The native transport does not have this limitation because when uploading checksums it uses a PutTaskEntity that is retryable because the PutTask that it wraps around is essentially a stream supplier. Here<https://github.com/apache/maven-resolver/blob/89dadad860dc802fd328f2c7a29e7285b599ecde/maven-resolver-transport-http/src/main/java/org/eclipse/aether/transport/http/HttpTransporter.java#L635>. For the native transport my colleagues think the issue is that it only supports retry when the client detects the closed connection before or while sending the request. But if the client sends the request and then detects the closed connection while trying to read the response, then it won't retry. In particular the native transport uses DefaultHttpRequestRetryHandler with requestSentRetryEnabled set to false (see here<https://github.com/apache/maven-resolver/blob/d0f8564846863c8a88f2fb1da8391fc860517192/maven-resolver-transport-http/src/main/java/org/eclipse/aether/transport/http/HttpTransporter.java#L239>). One way to fix the native transport is to change requestSentRetryEnabled to true when constructing DefaultHttpRequestRetryHandler. Another option is to use StandardHttpRequestRetryHandler instead because it treats PUT requests as idempotent so will retry a PUT request even when requestSentRetryEnabled is false. Oddly, if only one checksum algorithm is selected by using -Daether.checksums.algorithms=MD5 then the entire build fails on this error but if two or more algorithms are selected -Daether.checksums.algorithms=MD5,SHA-1 each checksum throws a warning but the overall build succeeds. It seems odd that the behavior is inconsistent here. I would expect it either to always throw a warning if a requested checksum failed upload, or to always fail the build if any checksum failed upload. The behavior with multiple checksums requested makes it clear that this failed to respond error can be caught by mvn. Using -Daether.connector.http.reuseConnections=false makes this error go away, because maven never tries sending data over an existing TCP connection. I am new to the maven dev community and was not sure what the best practice was for reporting these kinds of issues. Starting a conversation on the mailing list seems like a good place to start. What should happen next? Which of these issues deserve tickets? What more information should I provide and where? What PRs would be appreciated to help fix the issues? A stack trace from running "mvn deploy" with Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c) org.apache.http.NoHttpResponseException: ...:443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead (DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead (DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse (AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader (DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader (CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse (HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute (HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute (MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute (ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute (RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute (RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:72) at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:393) at org.eclipse.aether.transport.http.HttpTransporter.implPut (HttpTransporter.java:377) at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put (AbstractTransporter.java:107) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:608) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:591) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:565) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:414) at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:319) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:298) at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:201) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:392) at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:139) at org.apache.maven.plugins.deploy.DeployMojo.execute (DeployMojo.java:191) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175) at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76) at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163) at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283) at org.apache.maven.cli.MavenCli.main (MavenCli.java:206) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:566) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
