[
https://issues.apache.org/jira/browse/SOLR-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891108#comment-17891108
]
Sanjay Dutt commented on SOLR-17502:
------------------------------------
I have not tested it, but I believe sending "-http2" would send a special
header, and If we look at the code of HttpSolrCall, remoteQuery basically
copies headers from one request to another. That could cause it.
> request forwarding does not work with http/2
> --------------------------------------------
>
> Key: SOLR-17502
> URL: https://issues.apache.org/jira/browse/SOLR-17502
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 9.7
> Reporter: Sebastian Hofmann
> Priority: Major
>
> It seems like that requests using http/2 to solr/\{core}/config/overlay
> (maybe also others, i only tested these) which are forwarded from one Solr
> node to another in a cloud setup don´t work.
>
> The exact error message looks like this:
> {code:java}
> 2024-10-17 10:26:05.536 ERROR (qtp1677458155-20-null-79) [c: s: r: x:
> t:null-79] o.a.s.s.HttpSolrCall 500 Exception =>
> org.apache.solr.common.SolrException: Error trying to proxy request for url:
> http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817)
> solr-1 | org.apache.solr.common.SolrException: Error trying to proxy
> request for url: http://172.20.0.3:8983/solr/Test2/config/overlay with
> _forwardCount: 1
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817) ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.sendRemoteQuery(HttpSolrCall.java:504)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:251)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:208)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:243)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:213)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
> ~[?:?]
> solr-1 | at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
> ~[jetty-security-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> ~[jetty-servlet-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
> ~[jetty-rewrite-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Connection.offerTask(HTTP2Connection.java:193)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:144)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:97)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1136)
> ~[http2-common-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerSession.onFrame(HTTP2ServerSession.java:188)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http2.server.HTTP2ServerConnection.onOpen(HTTP2ServerConnection.java:113)
> ~[http2-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:424)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:455)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannelOverHttp.upgrade(HttpChannelOverHttp.java:677)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpChannelOverHttp.headerComplete(HttpChannelOverHttp.java:416)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1256)
> ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1542)
> ~[jetty-http-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:403)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:275)
> ~[jetty-server-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> ~[jetty-io-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
> ~[jetty-util-10.0.22.jar:10.0.22]
> solr-1 | at java.base/java.lang.Thread.run(Unknown Source) [?:?]
> solr-1 | Caused by: org.apache.http.client.ClientProtocolException
> solr-1 | at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
> solr-1 | Caused by: org.apache.http.ProtocolException: The server failed to
> respond with a valid HTTP response
> solr-1 | at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
> ~[?:?]
> solr-1 | at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
> ~[?:?]
> solr-1 | at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> ~[?:?]
> solr-1 | at
> org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:144)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
> solr-1 | at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> ~[?:?]
> solr-1 | at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[?:?]
> solr-1 | at
> org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
> solr-1 | ... 70 more
> {code}
> I created a small docker-compose.yml file to reproduce it:
> {code:java}
> version: '3'
> services:
> solr:
> image: solr:9.7
> ports:
> - "8983:8983"
> - "5005:5005"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> solr2:
> image: solr:9.7
> ports:
> - "8984:8983"
> environment:
> ZK_HOST: "zoo:2181"
> depends_on: [zoo]
> zoo:
> image: zookeeper:3.9
> ports:
> - "2181:2181"
> environment:
> ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok" {code}
> I created two collections using this curl commands:
> {code:java}
> curl
> "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
> curl
> "http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test2&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
> {code}
> In my case the collections were stored each on one node. I can send requests
> and solr forwards them to the right node:
> {code:java}
> curl "http://localhost:8983/solr/Test/config/overlay";
> {
> "responseHeader":{
> "status":0,
> "QTime":28
> },
> "overlay":{
> "znodeVersion":-1
> }
> }%
> curl "http://localhost:8983/solr/Test2/config/overlay";
> {
> "responseHeader":{
> "status":0,
> "QTime":25
> },
> "overlay":{
> "znodeVersion":-1
> }
> }% {code}
> Everything works fine, untill http/2 comes in to play and the server needs to
> forward the request, then the Request fails:
> {code:java}
> curl --http2 "http://localhost:8983/solr/Test2/config/overlay";
> curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
> curl --http2 "http://localhost:8983/solr/Test/config/overlay"
> {
> "responseHeader":{
> "status":0,
> "QTime":2
> },
> "overlay":{
> "znodeVersion":-1
> }
> }
> curl --http2 "http://localhost:8984/solr/Test/config/overlay";
> curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
> curl --http2 "http://localhost:8984/solr/Test2/config/overlay";
> {
> "responseHeader":{
> "status":0,
> "QTime":3
> },
> "overlay":{
> "znodeVersion":-1
> }
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
