Sebastian Hofmann created SOLR-17502:
----------------------------------------
Summary: overlay request forwarding does not work with http/2
Key: SOLR-17502
URL: https://issues.apache.org/jira/browse/SOLR-17502
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Affects Versions: 9.7
Reporter: Sebastian Hofmann
It seems like that requests using http/2 to solr/\{core}/config/overlay (maybe
also others, i only tested these) which are forwarded from one Solr node to
another in a cloud setup don´t work.
The exact error message looks like this:
{code:java}
2024-10-17 10:26:05.536 ERROR (qtp1677458155-20-null-79) [c: s: r: x:
t:null-79] o.a.s.s.HttpSolrCall 500 Exception =>
org.apache.solr.common.SolrException: Error trying to proxy request for url:
http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817)
solr-1 | org.apache.solr.common.SolrException: Error trying to proxy request
for url: http://172.20.0.3:8983/solr/Test2/config/overlay with _forwardCount: 1
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:817) ~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.sendRemoteQuery(HttpSolrCall.java:504)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:251)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:208)
~[?:?]
solr-1 | at
org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:243)
~[?:?]
solr-1 | at
org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:213)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
~[?:?]
solr-1 | at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
~[?:?]
solr-1 | at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
~[jetty-security-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
~[jetty-servlet-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
~[jetty-rewrite-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Connection.offerTask(HTTP2Connection.java:193)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnection.onNewStream(HTTP2ServerConnection.java:144)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onNewStream(HTTP2ServerConnectionFactory.java:97)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.HTTP2Session.notifyNewStream(HTTP2Session.java:1136)
~[http2-common-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerSession.onHeaders(HTTP2ServerSession.java:120)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerSession.onFrame(HTTP2ServerSession.java:188)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http2.server.HTTP2ServerConnection.onOpen(HTTP2ServerConnection.java:113)
~[http2-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:451)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.upgrade(HttpConnection.java:424)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.onCompleted(HttpConnection.java:455)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannelOverHttp.upgrade(HttpChannelOverHttp.java:677)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpChannelOverHttp.headerComplete(HttpChannelOverHttp.java:416)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1256)
~[jetty-http-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1542)
~[jetty-http-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:403)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:275)
~[jetty-server-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
~[jetty-io-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
~[jetty-util-10.0.22.jar:10.0.22]
solr-1 | at java.base/java.lang.Thread.run(Unknown Source) [?:?]
solr-1 | Caused by: org.apache.http.client.ClientProtocolException
solr-1 | at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
solr-1 | ... 70 more
solr-1 | Caused by: org.apache.http.ProtocolException: The server failed to
respond with a valid HTTP response
solr-1 | at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149)
~[?:?]
solr-1 | at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[?:?]
solr-1 | at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[?:?]
solr-1 | at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[?:?]
solr-1 | at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[?:?]
solr-1 | at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[?:?]
solr-1 | at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[?:?]
solr-1 | at
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:144)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[?:?]
solr-1 | at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
solr-1 | at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[?:?]
solr-1 | at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[?:?]
solr-1 | at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[?:?]
solr-1 | at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:780) ~[?:?]
solr-1 | ... 70 more
{code}
I created a small docker-compose.yml file to reproduce it:
{code:java}
version: '3'
services:
solr:
image: solr:9.7
ports:
- "8983:8983"
- "5005:5005"
environment:
ZK_HOST: "zoo:2181"
depends_on: [zoo]
solr2:
image: solr:9.7
ports:
- "8984:8983"
environment:
ZK_HOST: "zoo:2181"
depends_on: [zoo]
zoo:
image: zookeeper:3.9
ports:
- "2181:2181"
environment:
ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok" {code}
I created two collections using this curl commands:
{code:java}
curl
"http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
curl
"http://localhost:8983/solr/admin/collections?&action=CREATE&collection.configName=_default&name=Test2&numShards=1&replicationFactor=1&router.name=compositeId&wt=json";
{code}
In my case the collections were stored each on one node. I can send requests
and solr forwards them to the right node:
{code:java}
curl "http://localhost:8983/solr/Test/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":28
},
"overlay":{
"znodeVersion":-1
}
}%
curl "http://localhost:8983/solr/Test2/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":25
},
"overlay":{
"znodeVersion":-1
}
}% {code}
Everything works fine, untill http/2 comes in to play and the server needs to
forward the request, then the Request fails:
{code:java}
curl --http2 "http://localhost:8983/solr/Test2/config/overlay";
curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
curl --http2 "http://localhost:8983/solr/Test/config/overlay"
{
"responseHeader":{
"status":0,
"QTime":2
},
"overlay":{
"znodeVersion":-1
}
}
curl --http2 "http://localhost:8984/solr/Test/config/overlay";
curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)
curl --http2 "http://localhost:8984/solr/Test2/config/overlay";
{
"responseHeader":{
"status":0,
"QTime":3
},
"overlay":{
"znodeVersion":-1
}
}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
