Michael Welsh Duggan <m...@md5i.com> writes:
> Using Debian, dovecot-solr 1:2.2.26.0-4, and solr-tomcat 3.6.2+dfsg-9, I
> am getting 400 errors when doing searches. Here is an example search
> query from dovecot that failed (captured with wireshark):
>
> Frame 23: 338 bytes on wire (2704 bits), 338 bytes captured (2704 bits) on
> interface 0
> Linux cooked capture
> Internet Protocol Version 6, Src: ::1, Dst: ::1
> Transmission Control Protocol, Src Port: 56860, Dst Port: 8080, Seq: 1,
> Ack: 1, Len: 250
> Hypertext Transfer Protocol
> GET
> /solr/select?fl=uid,score&rows=2664&sort=uid+asc&q={!lucene+q.op%3dAND}(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i
> HTTP/1.1\r\n
> Host: localhost:8080\r\n
> Date: Mon, 19 Dec 2016 00:25:56 GMT\r\n
> Connection: Keep-Alive\r\n
> \r\n
> [Full request URI:
> http://localhost:8080/solr/select?fl=uid,score&rows=2664&sort=uid+asc&q={!lucene+q.op%3dAND}(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i]
> [HTTP request 1/1]
> [Response in frame: 25]
>
> Here is the same query from firefox, which succeeds:
>
> Frame 66: 646 bytes on wire (5168 bits), 646 bytes captured (5168 bits) on
> interface 0
> Linux cooked capture
> Internet Protocol Version 6, Src: ::1, Dst: ::1
> Transmission Control Protocol, Src Port: 56862, Dst Port: 8080, Seq: 1,
> Ack: 1, Len: 558
> Hypertext Transfer Protocol
> GET
> /solr/select?fl=uid,score&rows=2664&sort=uid+asc&q=%7B!lucene+q.op%3DAND%7D(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i
> HTTP/1.1\r\n
> Host: localhost:8080\r\n
> Connection: keep-alive\r\n
> Cache-Control: max-age=0\r\n
> Upgrade-Insecure-Requests: 1\r\n
> User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
> like Gecko) Chrome/55.0.2883.75 Safari/537.36\r\n
> Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n
> DNT: 1\r\n
> Accept-Encoding: gzip, deflate, sdch, br\r\n
> Accept-Language: en-US,en;q=0.8\r\n
> \r\n
> [Full request URI:
> http://localhost:8080/solr/select?fl=uid,score&rows=2664&sort=uid+asc&q=%7B!lucene+q.op%3DAND%7D(hdr:test+OR+body:test)&fq=%2Bbox:6d5de009f991854df726000012cf7b9c+%2Buser:md5i]
> [HTTP request 1/1]
> [Response in frame: 86]
>
>
> The salient difference seems to be the encoding of the braces. Indeed
> in the tomcat 8 logs, I find the following which seems to corroborate
> my hypothesis:
>
> java.lang.IllegalArgumentException: Invalid character found in the request
> target. The valid characters are defined in RFC 7230 and RFC 3986
> at
> org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:467)
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>
> Indeed the braces are listed in fts-backend-solr.c as part of
> solr_escape_chars, so don't know how the braces are making it through
> unencoded.
I have attached a patch which solves this problem. I initially tried
changing http_url_escape_param() to include braces, but this did not
solve the problem. I have to guess that the {!lucene+q.op=AND} bit does
not travel through this function. So I just changed the braces in the
lines where they were introduced into their encoded values. Since the
equals-sign was already encoded this way there, it seemed to make sense.
--- a/src/plugins/fts-solr/fts-backend-solr.c 2016-12-03 12:13:21.000000000 -0500
+++ b/src/plugins/fts-solr/fts-backend-solr.c 2017-01-16 14:28:11.000000000 -0500
@@ -828,7 +828,7 @@
mailbox_get_open_status(box, STATUS_UIDNEXT, &status);
str = t_str_new(256);
- str_printfa(str, "fl=uid,score&rows=%u&sort=uid+asc&q={!lucene+q.op%%3dAND}",
+ str_printfa(str, "fl=uid,score&rows=%u&sort=uid+asc&q=%%7b!lucene+q.op%%3dAND%%7d",
status.uidnext);
prefix_len = str_len(str);
@@ -937,7 +937,7 @@
string_t *str;
str = t_str_new(256);
- str_printfa(str, "fl=box,uid,score&rows=%u&sort=box+asc,uid+asc&q={!lucene+q.op%%3dAND}",
+ str_printfa(str, "fl=box,uid,score&rows=%u&sort=box+asc,uid+asc&q=%%7b!lucene+q.op%%3dAND%%7d",
SOLR_MAX_MULTI_ROWS);
if (solr_add_definite_query_args(str, args, and_args)) {
--
Michael Welsh Duggan
(m...@md5i.com)
