That is invalid JSON. The client needs to fix it. I’m surprised it indexes at all. This should not be your problem.
Past that string into this: https://jsonlint.com wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 28, 2022, at 12:57 PM, Matthew Castrigno <castr...@slhs.org> wrote: > > Hello Mikhail, > > I have to work with the payload as is, I cannot modify it. My entire solution > has a lot of other things going on which would just confuse the discussion. > > The issue I am having can be recreated using the update handler with the > script enabled (as shown in the documentation example) and json.command set > to false. > > Solr does not recognize a field with escape characters "\" > > Here is a much smaller payload that demonstrates the issue: > > from the script: > doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument > logger.warn(doc.toString()); > > sending this payload: > > {"partner":"88027688-62c4-459a-b4d5-a8ecf9edd1bf","command":"add","doc_id":"2ff99d1a-a21b-4391-9c47-af2865acb753","content":"Page\"} > > results in this output in the console, notice the "content" field is not > listed. Solr cannot parse this part of the payload, it simply ignores it. > > SolrInputDocument(fields: > [partner=88027688-62c4-459a-b4d5-a8ecf9edd1bf,​ command=add,​ > doc_id=2ff99d1a-a21b-4391-9c47-af2865acb753]) > > I am trying to find a way to filter out these escape characters so Solr, > specificaly, org.apache.solr.common.SolrInputDocument, will recognize the > fields that have them. > > Thank you. > ________________________________ > From: Mikhail Khludnev <m...@apache.org> > Sent: Monday, November 28, 2022 1:07 PM > To: users@solr.apache.org <users@solr.apache.org> > Subject: Re: Is there a way to run the entire payload of a request through a > charFilter and not just the fields? > > Hello, It's still not clear. Which update request params (or curl) you use? > What if you put content as a tiny string, and then complicate it step by > step? On Mon, Nov 28, 2022 at 7: 27 PM Matthew Castrigno <castrigm@ slhs. > org> wrote: > > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside the St. Luke's email system. > > ZjQcmQRYFpfptBannerEnd > > Hello, > It's still not clear. Which update request params (or curl) you use? What > if you put content as a tiny string, and then complicate it step by step? > > On Mon, Nov 28, 2022 at 7:27 PM Matthew Castrigno <castr...@slhs.org> wrote: > >> Hi Mikhail, >> >> Thank you for your response. I am currently using the script update >> processor, but I have not been able to access the entire payload for >> processing. cmd.solrDoc is not correctly reading the payload. I have a >> payload where it is not recognizing a field value. This payload had four >> fields, the last one is "content" but if I do this: >> doc = cmd.solrDoc; >> logger.warn(doc.toString()); >> The content field is not shown. >> >> I want to filter that field an remove the quotes, so it is recognized as >> additional JSON for me to process. >> >> logger output: >> SolrInputDocument(fields: >> [partner=88027688-62c4-459a-b4d5-a8ecf9edd1bf,​ command=add,​ >> doc_id=2ff99d1a-a21b-4391-9c47-af2865acb753]) >> Here is the payload: >> { >> "partner": "88027688-62c4-459a-b4d5-a8ecf9edd1bf", >> "command": "add", >> "doc_id": "2ff99d1a-a21b-4391-9c47-af2865acb753", >> "content": >> "{\"Page\":{\"Id\":\"2ff99d1a-a21b-4391-9c47-af2865acb753\",\"Name\":\"Ronald >> McDonald House Idaho >> meals\",\"Url\":\"/blogs/st-lukes/news-and-community/2021/jan/ronald-mcdonald-house-idaho-meals\",\"Date\":\"2022-10-03T12:30:17.3388537\",\"ContentType\":\"Blog\",\"Body\":{\"Fields\":[{\"Name\":\"Heading >> Background Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile Wide >> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Specialties\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Blog >> Post Name\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald >> House, St. Luke’s Children’s find new ways to help >> families\"},{\"Name\":\"Blog Summary\",\"Type\":\"Rich >> Text\",\"Value\":\"\"},{\"Name\":\"Share Summary\",\"Type\":\"Multi-Line >> Text\",\"Value\":\"\"},{\"Name\":\"ShareTitle\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >> ways to help families\"},{\"Name\":\"Blog Post >> Date\",\"Type\":\"Datetime\",\"Value\":\"2021-01-18T10:10:00Z\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Better Together\"},{\"Name\":\"Rss >> Link\",\"Type\":\"General >> Link\",\"Value\":\"\"},{\"Name\":\"Providers\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Main >> Blog Image Caption\",\"Type\":\"Single-Line >> Text\",\"Value\":\"\"},{\"Name\":\"Procedures and >> Treatments\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Special >> Services\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Page >> Title\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald House, >> St. Luke’s Children’s finding new ways to help >> families\"},{\"Name\":\"ShareImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Share >> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Channels\",\"Type\":\"Multilist\",\"Value\":\"Better >> Together\"},{\"Name\":\"Blog >> Tags\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileHeadline\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >> ways to help families\"},{\"Name\":\"Include in >> Sitemap\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Facilities\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile >> Category\",\"Type\":\"Droptree\",\"Value\":\"Blog >> Post\"},{\"Name\":\"BreadcrumbTitle\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >> ways to help >> families\"},{\"Name\":\"Author\",\"Type\":\"Droplink\",\"Value\":\"{E9CF1FC9-EF41-4B6F-9D78-F206A5997A84}\"},{\"Name\":\"Restricted >> To Pages\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Meta >> Keywords\",\"Type\":\"Single-Line >> Text\",\"Value\":\"\"},{\"Name\":\"Associated Content >> Type\",\"Type\":\"Droptree\",\"Value\":\"Blog Post\"},{\"Name\":\"Main Blog >> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"TileSummary\",\"Type\":\"Rich >> Text\",\"Value\":\"\"},{\"Name\":\"Meta Description\",\"Type\":\"Multi-Line >> Text\",\"Value\":\"\"},{\"Name\":\"Health >> Topics\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Icon\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"NavigationTitle\",\"Type\":\"Single-Line >> Text\",\"Value\":\"St. Luke’s Blogs\"},{\"Name\":\"Include in Search >> Index\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Conditions\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Heading >> Sub Text\",\"Type\":\"Single-Line Text\",\"Value\":\"Highlights from St. >> Luke’s and our community partners to improve >> health.\"},{\"Name\":\"typeaheadRollupCat\",\"Type\":\"Single-Line >> Text\",\"Value\":\"\"},{\"Name\":\"BlogPostYear\",\"Type\":\"Single-Line >> Text\",\"Value\":\"2021\"},{\"Name\":\"AuthorName\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Anna >> Fritz\"},{\"Name\":\"BlogCategory\",\"Type\":\"Single-Line >> Text\",\"Value\":\"News and >> Community\"}],\"Modules\":{\"Fields\":[{\"Name\":\"Content\",\"Type\":\"Rich >> Text\",\"Value\":\"\"},{\"Name\":\"Image >> Position\",\"Type\":\"Droptree\",\"Value\":\"Right\"},{\"Name\":\"Image >> Source\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Image >> Content\",\"Type\":\"Rich Text\",\"Value\":\"<p>For more than three >> decades, Ronald McDonald House Charities of Idaho has provided housing to >> families with children seeking medical care.</p>\\n<p>It also has found new >> ways to help during the COVID-19 era. </p>\\n<p>When the novel coronavirus >> gained a foothold in Idaho in March 2020, the organization had to put >> safety first and made the tough decision to temporarily stop accepting new >> families not already staying at its new Boise house. Instead, the >> organization paid for hotel rooms for families it could not >> accommodate. </p>\\n<p>“When everything happened, because we had to >> pull back services, we were trying to look for other ways we could help >> families,” said Taylor Munson, communications manager at Ronald McDonald >> House Charities of Idaho. </p>\\n<p>“They are obviously already in a >> stressful situation with a sick child, but the pandemic amplified that >> because there is even more unknown now.”</p>\\n<p>So, how could their staff >> keep serving families with kids in need? </p>\\n<p>The team at RMHCI >> decided to start assembling lunch boxes filled with meals for families with >> kids at St. Luke’s Children’s Hospital. </p>\\n<p>Since March, the staff >> has provided 4,770 meals to families at St. Luke’s.</p>\\n<p>“The lunches >> provided by the Ronald McDonald House have been a true blessing for our >> families in pediatrics, the pediatric ICU and the newborn ICU,” said Sherry >> Iverson, director of patient and family services at St. Luke’s >> Children’s.</p>\\n<p>“Being at the bedside of their children of all ages is >> top priority for parents and remembering to take care of themselves is >> easily forgotten. These lunches carefully assembled by the Ronald McDonald >> team and then delivered to their room provide a break, healthy food and a >> chance to reenergize during a very stressful time.” </p>\\n<p>An additional >> 920 meals have been provided by RMHCI to families with children receiving >> care at Saint Alphonsus Health System. </p>\\n<p>The Ronald McDonald staff >> provides the lunch boxes four days a week, typically including sandwiches, >> fruit and chips, as well as snack bags. The total cost of the lunches so >> far has been about $24,000.</p>\\n<p>“Without these wonderful care >> packages, many parents would go all day without food,” Iverson said. “This >> partnership has been so important during this COVID >> pandemic.”</p>\\n<p>Some of the food items are donated from local >> organizations, while others are purchased by staff and assembled in the >> kitchen at the new Ronald McDonald House facility, near the St. Luke’s >> Boise Medical Center. </p>\\n<p>St. Luke’s employees pick up the meals and >> take them across the street to the children’s hospital.</p>\\n<p>“The >> feedback that we’ve gotten from families and nurses and people over at St. >> Luke’s is that it’s so helpful because families either may not have money >> to get food or they don’t want to leave their child’s bedside,” Munson >> said.</p>\\n<p>The Ronald McDonald House started accepting new families >> again at its facility in May 2020. St. Luke’s Children’s Hospital is the >> only children’s hospital in Idaho, which has led to a strong partnership >> between the medical center and nearby Ronald McDonald House. </p>\\n<p>“It >> has been very collaborative with St. Luke’s. We wanted to make sure what we >> were going to be doing was beneficial,” Munson said. </p>\\n<p>The program >> will continue through the end of March, marking one full year of providing >> meals, and then the Ronald McDonald House staff will reevaluate for short- >> and long-term plans, Munson said. </p>\\n<p>“The pandemic obviously isn’t >> ideal, but it did allow us find new ways of helping families,” Munson said. >> “A lot of our focus is family centered care—that’s really our goal, and >> feeding families is a big part of >> that.”</p>\\n<p><br>\\n</p>\"},{\"Name\":\"Channel\",\"Type\":\"Droptree\",\"Value\":\"Better >> Together\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Better Together\"},{\"Name\":\"Number to >> Display\",\"Type\":\"Integer\",\"Value\":\"4\"},{\"Name\":\"Related >> Item\",\"Type\":\"Droptree\",\"Value\":\"St Lukes Childrens >> Hospital\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >> Text\",\"Value\":\"Related >> Hospital\"}]}},\"Facets\":[\"Blogs\",\"Article\"],\"Title\":\"Ronald >> McDonald House, St. Luke’s Children’s find new ways to help >> families\",\"Summary\":\"\"}}" >> } >> >> ________________________________ >> From: Mikhail Khludnev <m...@apache.org> >> Sent: Saturday, November 26, 2022 1:28 PM >> To: users@solr.apache.org <users@solr.apache.org> >> Subject: Re: Is there a way to run the entire payload of a request through >> a charFilter and not just the fields? >> >> Hi Matthew. Can it be https: //urldefense. com/v3/__https: //solr. apache. >> org/guide/solr/latest/configuration-guide/script-update-processor. >> html__;!!FkC3_z_N!ON6B9iNNwK7AkdwAKGpLzLAzNKXR4m8SIom95HENXZNK381f6vhLlbAf5l7Z2mpVNUNJWAP2dw$ >> ? On Sat, >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside the St. Luke's email system. >> >> ZjQcmQRYFpfptBannerEnd >> >> Hi Matthew. >> Can it be >> >> https://urldefense.com/v3/__https://solr.apache.org/guide/solr/latest/configuration-guide/script-update-processor.html__;!!FkC3_z_N!ON6B9iNNwK7AkdwAKGpLzLAzNKXR4m8SIom95HENXZNK381f6vhLlbAf5l7Z2mpVNUNJWAP2dw$ >> ? >> >> On Sat, Nov 26, 2022 at 1:15 AM Matthew Castrigno <castr...@slhs.org> >> wrote: >> >>> I need to filter out some characters in a payload so that SOLR will >>> recognize the payload as a JSON document. >>> >>> The solr.MappingCharFilterFactory functionality is what I need but I need >>> to run over the entire payload and not just the fields. >>> >>> I cannot change the payload prior to submitting to SOLR. >>> >>> Is there any way to accomplish this? >>> >>> Any insights are most appreciated. >>> >>> Thank you. >>> >>> ---------------------------------------------------------------------- >>> "This message is intended for the use of the person or entity to which it >>> is addressed and may contain information that is confidential or >>> privileged, the disclosure of which is governed by applicable law. If the >>> reader of this message is not the intended recipient, you are hereby >>> notified that any dissemination, distribution, or copying of this >>> information is strictly prohibited. If you have received this message by >>> error, please notify us immediately and destroy the related message." >>> >> >> >> -- >> Sincerely yours >> Mikhail Khludnev >> >> >> ---------------------------------------------------------------------- >> "This message is intended for the use of the person or entity to which it >> is addressed and may contain information that is confidential or >> privileged, the disclosure of which is governed by applicable law. If the >> reader of this message is not the intended recipient, you are hereby >> notified that any dissemination, distribution, or copying of this >> information is strictly prohibited. If you have received this message by >> error, please notify us immediately and destroy the related message." >> > > > -- > Sincerely yours > Mikhail Khludnev > > > ---------------------------------------------------------------------- > "This message is intended for the use of the person or entity to which it is > addressed and may contain information that is confidential or privileged, the > disclosure of which is governed by applicable law. If the reader of this > message is not the intended recipient, you are hereby notified that any > dissemination, distribution, or copying of this information is strictly > prohibited. If you have received this message by error, please notify us > immediately and destroy the related message."
