I think I would write an update request processor script, take the value of the “content” field, then parse that as JSON, then take those fields and add them.
https://solr.apache.org/guide/solr/latest/configuration-guide/script-update-processor.html The update request processor scripts are a little weird to program, because you use JavaScript syntax but call Java libraries, then you need to debug with output into the Solr log. But once you get them working, they are really handy. I would also make sure that the request is being interpreted as a JSON document and not as Solr update format. https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html#json-formatted-index-updates wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 28, 2022, at 2:59 PM, Matthew Castrigno <castr...@slhs.org> wrote: > > Thank you for your reply Walter. When I say "payload" I mean what you would > put in the raw data of a curl request. I am pretty sure SOLR would refer to > this as the "document" > > To the question: > "You want to parse that JSON and treat it as additional fields to index?" > > Yes, and it requires further processing with my script to determine the > fields to add. > > Since my last message, I made an additional discovery, my question now > becomes something quite different. I had avoided other details as to not > confuse the issue, specifically parameters that I was using. It appears that > they are relevant. I was able to get expected output to the logger by > commenting out some of default parameters I had configured. This is going to > get a little confusing, so please bear with me. > > Inside the "content" field is a json object that I want to flatten once the > escape characters are removed. > > It appears that the field name "content" is somehow interacting with field > parameters that have "content" in the path. If I comment them out then the > output of > > doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument > logger.warn(doc.toString()); > > is as expected with content field having a long string with escape characters > in it. > > So now it seems my issue is how keep these parameters from interfering with > org.apache.solr.common.SolrInputDocument so I can access the "content" field. > > It would seem I now have to take a different approach. Remove these > parameters, take in the unwieldly "content" field values and flatten and > process it's contents myself. > > > Once other side question is what are all these "& #8203" coming from and why > does solr put them in there? > > From my solrconfig: > <requestHandler name="/index" class="solr.UpdateRequestHandler"> > <lst name="defaults"> > <str name="commit">true</str> > <str name="json.command">false</str> > <str name="split">/</str> > <str name="update.chain">script</str> > <str name="f">$FQN:/**</str> > <str name="f">docId_s:/content/Page/Id</str> > <str name="f">Name_s:/content/Page/Name</str> > <str name="f">url:/content/Page/Url</str> > <str name="f">date_dt:/content/Page/Date</str> > <str name="f">contenttype:/content/Page/ContentType</str> > <str name="f">title:/content/Page/Title</str> > <str name="f">summary_s:/content/Page/Summary</str> > <str name="f">facets_ss:/content/Page/Facets</str> > </lst> > > Below is the curl request and then the logger console output. > > > curl --location --request GET > 'http://solrztestdb01.sl1.stlukes-int.org:8983/solr/talix/index' \ > --header 'Content-Type: application/json' \ > --data-raw > '{"partner":"88027688-62c4-459a-b4d5-a8ecf9edd1bf","command":"add","doc_id":"2ff99d1a-a21b-4391-9c47-af2865acb753","content":"{\"Page\":{\"Id\":\"2ff99d1a-a21b-4391-9c47-af2865acb753\",\"Name\":\"Ronald > McDonald House Idaho > meals\",\"Url\":\"/blogs/st-lukes/news-and-community/2021/jan/ronald-mcdonald-house-idaho-meals\",\"Date\":\"2022-10-03T12:30:17.3388537\",\"ContentType\":\"Blog\",\"Body\":{\"Fields\":[{\"Name\":\"Heading > Background Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile Wide > Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Specialties\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Blog > Post Name\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald House, > St. Luke’s Children’s find new ways to help families\"},{\"Name\":\"Blog > Summary\",\"Type\":\"Rich Text\",\"Value\":\"\"},{\"Name\":\"Share > Summary\",\"Type\":\"Multi-Line > Text\",\"Value\":\"\"},{\"Name\":\"ShareTitle\",\"Type\":\"Single-Line > Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new ways > to help families\"},{\"Name\":\"Blog Post > Date\",\"Type\":\"Datetime\",\"Value\":\"2021-01-18T10:10:00Z\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line > Text\",\"Value\":\"Better Together\"},{\"Name\":\"Rss > Link\",\"Type\":\"General > Link\",\"Value\":\"\"},{\"Name\":\"Providers\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Main > Blog Image Caption\",\"Type\":\"Single-Line > Text\",\"Value\":\"\"},{\"Name\":\"Procedures and > Treatments\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Special > Services\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Page > Title\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald House, St. > Luke’s Children’s finding new ways to help > families\"},{\"Name\":\"ShareImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Share > > Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Channels\",\"Type\":\"Multilist\",\"Value\":\"Better > Together\"},{\"Name\":\"Blog > Tags\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileHeadline\",\"Type\":\"Single-Line > Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new > ways to help families\"},{\"Name\":\"Include in > Sitemap\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Facilities\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile > Category\",\"Type\":\"Droptree\",\"Value\":\"Blog > Post\"},{\"Name\":\"BreadcrumbTitle\",\"Type\":\"Single-Line > Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new ways > to help > families\"},{\"Name\":\"Author\",\"Type\":\"Droplink\",\"Value\":\"{E9CF1FC9-EF41-4B6F-9D78-F206A5997A84}\"},{\"Name\":\"Restricted > To Pages\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Meta > Keywords\",\"Type\":\"Single-Line > Text\",\"Value\":\"\"},{\"Name\":\"Associated Content > Type\",\"Type\":\"Droptree\",\"Value\":\"Blog Post\"},{\"Name\":\"Main Blog > Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"TileSummary\",\"Type\":\"Rich > Text\",\"Value\":\"\"},{\"Name\":\"Meta Description\",\"Type\":\"Multi-Line > Text\",\"Value\":\"\"},{\"Name\":\"Health > Topics\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Icon\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"NavigationTitle\",\"Type\":\"Single-Line > Text\",\"Value\":\"St. Luke’s Blogs\"},{\"Name\":\"Include in Search > Index\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Conditions\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Heading > Sub Text\",\"Type\":\"Single-Line Text\",\"Value\":\"Highlights from St. > Luke’s and our community partners to improve > health.\"},{\"Name\":\"typeaheadRollupCat\",\"Type\":\"Single-Line > Text\",\"Value\":\"\"},{\"Name\":\"BlogPostYear\",\"Type\":\"Single-Line > Text\",\"Value\":\"2021\"},{\"Name\":\"AuthorName\",\"Type\":\"Single-Line > Text\",\"Value\":\"Anna > Fritz\"},{\"Name\":\"BlogCategory\",\"Type\":\"Single-Line > Text\",\"Value\":\"News and > Community\"}],\"Modules\":{\"Fields\":[{\"Name\":\"Content\",\"Type\":\"Rich > Text\",\"Value\":\"\"},{\"Name\":\"Image > Position\",\"Type\":\"Droptree\",\"Value\":\"Right\"},{\"Name\":\"Image > Source\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Image > Content\",\"Type\":\"Rich Text\",\"Value\":\"<p>For more than three decades, > Ronald McDonald House Charities of Idaho has provided housing to families > with children seeking medical care.</p>\\n<p>It also has found new ways to > help during the COVID-19 era. </p>\\n<p>When the novel coronavirus gained a > foothold in Idaho in March 2020, the organization had to put safety first and > made the tough decision to temporarily stop accepting new families not > already staying at its new Boise house. Instead, the organization paid for > hotel rooms for families it could not accommodate. </p>\\n<p>“When > everything happened, because we had to pull back services, we were trying to > look for other ways we could help families,” said Taylor Munson, > communications manager at Ronald McDonald House Charities of Idaho. > </p>\\n<p>“They are obviously already in a stressful situation with a sick > child, but the pandemic amplified that because there is even more unknown > now.”</p>\\n<p>So, how could their staff keep serving families with kids in > need? </p>\\n<p>The team at RMHCI decided to start assembling lunch boxes > filled with meals for families with kids at St. Luke’s Children’s Hospital. > </p>\\n<p>Since March, the staff has provided 4,770 meals to families at St. > Luke’s.</p>\\n<p>“The lunches provided by the Ronald McDonald House have been > a true blessing for our families in pediatrics, the pediatric ICU and the > newborn ICU,” said Sherry Iverson, director of patient and family services at > St. Luke’s Children’s.</p>\\n<p>“Being at the bedside of their children of > all ages is top priority for parents and remembering to take care of > themselves is easily forgotten. These lunches carefully assembled by the > Ronald McDonald team and then delivered to their room provide a break, > healthy food and a chance to reenergize during a very stressful time.” > </p>\\n<p>An additional 920 meals have been provided by RMHCI to families > with children receiving care at Saint Alphonsus Health System. </p>\\n<p>The > Ronald McDonald staff provides the lunch boxes four days a week, typically > including sandwiches, fruit and chips, as well as snack bags. The total cost > of the lunches so far has been about $24,000.</p>\\n<p>“Without these > wonderful care packages, many parents would go all day without food,” Iverson > said. “This partnership has been so important during this COVID > pandemic.”</p>\\n<p>Some of the food items are donated from local > organizations, while others are purchased by staff and assembled in the > kitchen at the new Ronald McDonald House facility, near the St. Luke’s Boise > Medical Center. </p>\\n<p>St. Luke’s employees pick up the meals and take > them across the street to the children’s hospital.</p>\\n<p>“The feedback > that we’ve gotten from families and nurses and people over at St. Luke’s is > that it’s so helpful because families either may not have money to get food > or they don’t want to leave their child’s bedside,” Munson said.</p>\\n<p>The > Ronald McDonald House started accepting new families again at its facility in > May 2020. St. Luke’s Children’s Hospital is the only children’s hospital in > Idaho, which has led to a strong partnership between the medical center and > nearby Ronald McDonald House. </p>\\n<p>“It has been very collaborative with > St. Luke’s. We wanted to make sure what we were going to be doing was > beneficial,” Munson said. </p>\\n<p>The program will continue through the end > of March, marking one full year of providing meals, and then the Ronald > McDonald House staff will reevaluate for short- and long-term plans, Munson > said. </p>\\n<p>“The pandemic obviously isn’t ideal, but it did allow us find > new ways of helping families,” Munson said. “A lot of our focus is family > centered care—that’s really our goal, and feeding families is a big part of > that.”</p>\\n<p><br>\\n</p>\"},{\"Name\":\"Channel\",\"Type\":\"Droptree\",\"Value\":\"Better > Together\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line > Text\",\"Value\":\"Better Together\"},{\"Name\":\"Number to > Display\",\"Type\":\"Integer\",\"Value\":\"4\"},{\"Name\":\"Related > Item\",\"Type\":\"Droptree\",\"Value\":\"St Lukes Childrens > Hospital\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line > Text\",\"Value\":\"Related > Hospital\"}]}},\"Facets\":[\"Blogs\",\"Article\"],\"Title\":\"Ronald McDonald > House, St. Luke’s Children’s find new ways to help > families\",\"Summary\":\"\"}}"}' > > > > SolrInputDocument(fields: > [partner=88027688-62c4-459a-b4d5-a8ecf9edd1bf,​ command=add,​ > doc_id=2ff99d1a-a21b-4391-9c47-af2865acb753,​ > content={"Page":{"Id":"2ff99d1a-a21b-4391-9c47-af2865acb753",​"Name":"Ronald > McDonald House Idaho > meals",​"Url":"/blogs/st-lukes/news-and-community/2021/jan/ronald-mcdonald-house-idaho-meals",​"Date":"2022-10-03T12:30:17.3388537",​"ContentType":"Blog",​"Body":{"Fields":[{"Name":"Heading > Background > Image",​"Type":"Image",​"Value":""},​{"Name":"Tile Wide > Image",​"Type":"Image",​"Value":""},​{"Name":"Specialties",​"Type":"Treelist",​"Value":""},​{"Name":"Blog > Post Name",​"Type":"Single-Line Text",​"Value":"Ronald McDonald > House,​ St. Luke’s Children’s find new ways to help > families"},​{"Name":"Blog Summary",​"Type":"Rich > Text",​"Value":""},​{"Name":"Share > Summary",​"Type":"Multi-Line > Text",​"Value":""},​{"Name":"ShareTitle",​"Type":"Single-Line > Text",​"Value":"Ronald McDonald House,​ St. Luke’s Children’s > find new ways to help families"},​{"Name":"Blog Post > Date",​"Type":"Datetime",​"Value":"2021-01-18T10:10:00Z"},​{"Name":"Heading",​"Type":"Single-Line > Text",​"Value":"Better Together"},​{"Name":"Rss > Link",​"Type":"General > Link",​"Value":""},​{"Name":"Providers",​"Type":"Treelist",​"Value":""},​{"Name":"Main > Blog Image Caption",​"Type":"Single-Line > Text",​"Value":""},​{"Name":"Procedures and > Treatments",​"Type":"Multilist",​"Value":""},​{"Name":"Special > Services",​"Type":"Treelist",​"Value":""},​{"Name":"Page > Title",​"Type":"Single-Line Text",​"Value":"Ronald McDonald > House,​ St. Luke’s Children’s finding new ways to help > families"},​{"Name":"ShareImage",​"Type":"Image",​"Value":""},​{"Name":"Share > > Image",​"Type":"Image",​"Value":""},​{"Name":"Channels",​"Type":"Multilist",​"Value":"Better > Together"},​{"Name":"Blog > Tags",​"Type":"Treelist",​"Value":""},​{"Name":"TileHeadline",​"Type":"Single-Line > Text",​"Value":"Ronald McDonald House,​ St. Luke’s Children’s > find new ways to help families"},​{"Name":"Include in > Sitemap",​"Type":"Checkbox",​"Value":"1"},​{"Name":"Facilities",​"Type":"Treelist",​"Value":""},​{"Name":"TileImage",​"Type":"Image",​"Value":""},​{"Name":"Tile > Category",​"Type":"Droptree",​"Value":"Blog > Post"},​{"Name":"BreadcrumbTitle",​"Type":"Single-Line > Text",​"Value":"Ronald McDonald House,​ St. Luke’s Children’s > find new ways to help > families"},​{"Name":"Author",​"Type":"Droplink",​"Value":"{E9CF1FC9-EF41-4B6F-9D78-F206A5997A84}"},​{"Name":"Restricted > To Pages",​"Type":"Treelist",​"Value":""},​{"Name":"Meta > Keywords",​"Type":"Single-Line > Text",​"Value":""},​{"Name":"Associated Content > Type",​"Type":"Droptree",​"Value":"Blog > Post"},​{"Name":"Main Blog > Image",​"Type":"Image",​"Value":""},​{"Name":"TileSummary",​"Type":"Rich > Text",​"Value":""},​{"Name":"Meta > Description",​"Type":"Multi-Line > Text",​"Value":""},​{"Name":"Health > Topics",​"Type":"Multilist",​"Value":""},​{"Name":"Icon",​"Type":"Image",​"Value":""},​{"Name":"NavigationTitle",​"Type":"Single-Line > Text",​"Value":"St. Luke’s Blogs"},​{"Name":"Include in Search > Index",​"Type":"Checkbox",​"Value":"1"},​{"Name":"Conditions",​"Type":"Treelist",​"Value":""},​{"Name":"Heading > Sub Text",​"Type":"Single-Line Text",​"Value":"Highlights from > St. Luke’s and our community partners to improve > health."},​{"Name":"typeaheadRollupCat",​"Type":"Single-Line > Text",​"Value":""},​{"Name":"BlogPostYear",​"Type":"Single-Line > > Text",​"Value":"2021"},​{"Name":"AuthorName",​"Type":"Single-Line > Text",​"Value":"Anna > Fritz"},​{"Name":"BlogCategory",​"Type":"Single-Line > Text",​"Value":"News and > Community"}],​"Modules":{"Fields":[{"Name":"Content",​"Type":"Rich > Text",​"Value":""},​{"Name":"Image > Position",​"Type":"Droptree",​"Value":"Right"},​{"Name":"Image > Source",​"Type":"Image",​"Value":""},​{"Name":"Image > Content",​"Type":"Rich Text",​"Value":"<p>For more than three > decades,​ Ronald McDonald House Charities of Idaho has provided housing > to families with children seeking medical care.</p>\n<p>It also has found new > ways to help during the COVID-19 era. </p>\n<p>When the novel coronavirus > gained a foothold in Idaho in March 2020,​ the organization had to put > safety first and made the tough decision to temporarily stop accepting new > families not already staying at its new Boise house. Instead,​ the > organization paid for hotel rooms for families it could not > accommodate. </p>\n<p>“When everything happened,​ because we had > to pull back services,​ we were trying to look for other ways we could > help families,​” said Taylor Munson,​ communications manager at > Ronald McDonald House Charities of Idaho. </p>\n<p>“They are obviously > already in a stressful situation with a sick child,​ but the pandemic > amplified that because there is even more unknown now.”</p>\n<p>So,​ > how could their staff keep serving families with kids in need? </p>\n<p>The > team at RMHCI decided to start assembling lunch boxes filled with meals for > families with kids at St. Luke’s Children’s Hospital. </p>\n<p>Since > March,​ the staff has provided 4,​770 meals to families at St. > Luke’s.</p>\n<p>“The lunches provided by the Ronald McDonald House have been > a true blessing for our families in pediatrics,​ the pediatric ICU and > the newborn ICU,​” said Sherry Iverson,​ director of patient and > family services at St. Luke’s Children’s.</p>\n<p>“Being at the bedside of > their children of all ages is top priority for parents and remembering to > take care of themselves is easily forgotten. These lunches carefully > assembled by the Ronald McDonald team and then delivered to their room > provide a break,​ healthy food and a chance to reenergize during a very > stressful time.” </p>\n<p>An additional 920 meals have been provided by RMHCI > to families with children receiving care at Saint Alphonsus Health System. > </p>\n<p>The Ronald McDonald staff provides the lunch boxes four days a > week,​ typically including sandwiches,​ fruit and chips,​ > as well as snack bags. The total cost of the lunches so far has been about > $24,​000.</p>\n<p>“Without these wonderful care packages,​ many > parents would go all day without food,​” Iverson said. “This > partnership has been so important during this COVID pandemic.”</p>\n<p>Some > of the food items are donated from local organizations,​ while others > are purchased by staff and assembled in the kitchen at the new Ronald > McDonald House facility,​ near the St. Luke’s Boise Medical Center. > </p>\n<p>St. Luke’s employees pick up the meals and take them across the > street to the children’s hospital.</p>\n<p>“The feedback that we’ve gotten > from families and nurses and people over at St. Luke’s is that it’s so > helpful because families either may not have money to get food or they don’t > want to leave their child’s bedside,​” Munson said.</p>\n<p>The Ronald > McDonald House started accepting new families again at its facility in May > 2020. St. Luke’s Children’s Hospital is the only children’s hospital in > Idaho,​ which has led to a strong partnership between the medical > center and nearby Ronald McDonald House. </p>\n<p>“It has been very > collaborative with St. Luke’s. We wanted to make sure what we were going to > be doing was beneficial,​” Munson said. </p>\n<p>The program will > continue through the end of March,​ marking one full year of providing > meals,​ and then the Ronald McDonald House staff will reevaluate for > short- and long-term plans,​ Munson said. </p>\n<p>“The pandemic > obviously isn’t ideal,​ but it did allow us find new ways of helping > families,​” Munson said. “A lot of our focus is family centered > care—that’s really our goal,​ and feeding families is a big part of > that.”</p>\n<p><br>\n</p>"},​{"Name":"Channel",​"Type":"Droptree",​"Value":"Better > Together"},​{"Name":"Heading",​"Type":"Single-Line > Text",​"Value":"Better Together"},​{"Name":"Number to > Display",​"Type":"Integer",​"Value":"4"},​{"Name":"Related > Item",​"Type":"Droptree",​"Value":"St Lukes Childrens > Hospital"},​{"Name":"Heading",​"Type":"Single-Line > Text",​"Value":"Related > Hospital"}]}},​"Facets":["Blogs",​"Article"],​"Title":"Ronald > McDonald House,​ St. Luke’s Children’s find new ways to help > families",​"Summary":""}}]) > ________________________________ > From: Walter Underwood <wun...@wunderwood.org> > Sent: Monday, November 28, 2022 3:03 PM > To: users@solr.apache.org <users@solr.apache.org> > Subject: Re: Is there a way to run the entire payload of a request through a > charFilter and not just the fields? > > I think the word “payload” is confusing me. The client is sending a JSON > document. That JSON document has a “content” field which is string-valued and > is escaped (stringified) JSON. Correct? You want to parse that JSON and treat > it as additional > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside the St. Luke's email system. > > ZjQcmQRYFpfptBannerEnd > > I think the word “payload” is confusing me. > > The client is sending a JSON document. That JSON document has a “content” > field which is string-valued and is escaped (stringified) JSON. Correct? > > You want to parse that JSON and treat it as additional fields to index? > > So this content (fragment): > > "content": > "{\"Page\":{\"Id\":\"2ff99d1a-a21b-4391-9c47-af2865acb753\",\"Name\":\"Ronald > McDonald House Idaho > meals\",\"Url\":\"/blogs/st-lukes/news-and-community/2021/jan/ronald-mcdonald-house-idaho-meals\",\"Date\":\"2022-10-03T12:30:17.3388537\",\"ContentType\":\"Blog\",\"Body\":{\"Fields\":[{\"Name\":\"Heading > Background Image\",\"Type\":\"Image\",\"Value\":\"\”},... > > would add fields like Page, and Id and Name under that? > > wunder > Walter Underwood > wun...@wunderwood.org > https://urldefense.com/v3/__http://observer.wunderwood.org/__;!!FkC3_z_N!IgyPCkR78yw2ghOvw5ATxnQzprL1__srg0aZ896fty4PqrGVImlbbzaO_MATxGOlxvUVrvuLPA0FU_dQ$ > (my blog) > >> On Nov 28, 2022, at 1:31 PM, Matthew Castrigno <castr...@slhs.org> wrote: >> >> Hello Walter, >> >> Thank you for your reply. Yes, it is invalid JSON. However, it is "my" >> problem unfortunately. >> >> I am looking for a way to filter the payload as a character string. >> >> The charFilter would be great, however for that to work it would have to be >> first recognized as a valid field. >> >> Is there a way in SOLR to process the entire payload in this way so I can >> turn it into proper JSON by filtering out the "/" s ? >> >> Thank you. >> From: Walter Underwood <wun...@wunderwood.org <mailto:wun...@wunderwood.org>> >> Sent: Monday, November 28, 2022 2:23 PM >> To: users@solr.apache.org <mailto:users@solr.apache.org> >> <users@solr.apache.org <mailto:users@solr.apache.org>> >> Subject: Re: Is there a way to run the entire payload of a request through a >> charFilter and not just the fields? >> >> This Message Is From an External Sender >> This message came from outside the St. Luke's email system. >> That is invalid JSON. The client needs to fix it. I’m surprised it indexes >> at all. This should not be your problem. >> >> Past that string into this: >> https://urldefense.com/v3/__https://jsonlint.com__;!!FkC3_z_N!K2Droj3x11Rpw7SE8FWYfCL5RP-Csp8j-9RRLv1EYypmATl8cteTtGrFKpxLPrxknF9jN0quzVuaiamt$<https://urldefense.com/v3/__https://jsonlint.com__;!!FkC3_z_N!K2Droj3x11Rpw7SE8FWYfCL5RP-Csp8j-9RRLv1EYypmATl8cteTtGrFKpxLPrxknF9jN0quzVuaiamt$> >> >> wunder >> Walter Underwood >> wun...@wunderwood.org <mailto:wun...@wunderwood.org> >> https://urldefense.com/v3/__http://observer.wunderwood.org/__;!!FkC3_z_N!K2Droj3x11Rpw7SE8FWYfCL5RP-Csp8j-9RRLv1EYypmATl8cteTtGrFKpxLPrxknF9jN0quzRPZK7sh$<https://urldefense.com/v3/__http://observer.wunderwood.org/__;!!FkC3_z_N!K2Droj3x11Rpw7SE8FWYfCL5RP-Csp8j-9RRLv1EYypmATl8cteTtGrFKpxLPrxknF9jN0quzRPZK7sh$> >> (my blog) >> >>> On Nov 28, 2022, at 12:57 PM, Matthew Castrigno <castr...@slhs.org >>> <mailto:castr...@slhs.org>> wrote: >>> >>> Hello Mikhail, >>> >>> I have to work with the payload as is, I cannot modify it. My entire >>> solution has a lot of other things going on which would just confuse the >>> discussion. >>> >>> The issue I am having can be recreated using the update handler with the >>> script enabled (as shown in the documentation example) and json.command >>> set to false. >>> >>> Solr does not recognize a field with escape characters "\" >>> >>> Here is a much smaller payload that demonstrates the issue: >>> >>> from the script: >>> doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument >>> logger.warn(doc.toString()); >>> >>> sending this payload: >>> >>> {"partner":"88027688-62c4-459a-b4d5-a8ecf9edd1bf","command":"add","doc_id":"2ff99d1a-a21b-4391-9c47-af2865acb753","content":"Page\"} >>> >>> results in this output in the console, notice the "content" field is not >>> listed. Solr cannot parse this part of the payload, it simply ignores it. >>> >>> SolrInputDocument(fields: >>> [partner=88027688-62c4-459a-b4d5-a8ecf9edd1bf,​ command=add,​ >>> doc_id=2ff99d1a-a21b-4391-9c47-af2865acb753]) >>> >>> I am trying to find a way to filter out these escape characters so Solr, >>> specificaly, org.apache.solr.common.SolrInputDocument, will recognize the >>> fields that have them. >>> >>> Thank you. >>> ________________________________ >>> From: Mikhail Khludnev <m...@apache.org <mailto:m...@apache.org>> >>> Sent: Monday, November 28, 2022 1:07 PM >>> To: users@solr.apache.org <mailto:users@solr.apache.org> >>> <users@solr.apache.org <mailto:users@solr.apache.org>> >>> Subject: Re: Is there a way to run the entire payload of a request through >>> a charFilter and not just the fields? >>> >>> Hello, It's still not clear. Which update request params (or curl) you use? >>> What if you put content as a tiny string, and then complicate it step by >>> step? On Mon, Nov 28, 2022 at 7: 27 PM Matthew Castrigno <castrigm@ slhs. >>> org> wrote: > >>> ZjQcmQRYFpfptBannerStart >>> This Message Is From an External Sender >>> This message came from outside the St. Luke's email system. >>> >>> ZjQcmQRYFpfptBannerEnd >>> >>> Hello, >>> It's still not clear. Which update request params (or curl) you use? What >>> if you put content as a tiny string, and then complicate it step by step? >>> >>> On Mon, Nov 28, 2022 at 7:27 PM Matthew Castrigno <castr...@slhs.org >>> <mailto:castr...@slhs.org>> wrote: >>> >>>> Hi Mikhail, >>>> >>>> Thank you for your response. I am currently using the script update >>>> processor, but I have not been able to access the entire payload for >>>> processing. cmd.solrDoc is not correctly reading the payload. I have a >>>> payload where it is not recognizing a field value. This payload had four >>>> fields, the last one is "content" but if I do this: >>>> doc = cmd.solrDoc; >>>> logger.warn(doc.toString()); >>>> The content field is not shown. >>>> >>>> I want to filter that field an remove the quotes, so it is recognized as >>>> additional JSON for me to process. >>>> >>>> logger output: >>>> SolrInputDocument(fields: >>>> [partner=88027688-62c4-459a-b4d5-a8ecf9edd1bf,​ command=add,​ >>>> doc_id=2ff99d1a-a21b-4391-9c47-af2865acb753]) >>>> Here is the payload: >>>> { >>>> "partner": "88027688-62c4-459a-b4d5-a8ecf9edd1bf", >>>> "command": "add", >>>> "doc_id": "2ff99d1a-a21b-4391-9c47-af2865acb753", >>>> "content": >>>> "{\"Page\":{\"Id\":\"2ff99d1a-a21b-4391-9c47-af2865acb753\",\"Name\":\"Ronald >>>> McDonald House Idaho >>>> meals\",\"Url\":\"/blogs/st-lukes/news-and-community/2021/jan/ronald-mcdonald-house-idaho-meals\",\"Date\":\"2022-10-03T12:30:17.3388537\",\"ContentType\":\"Blog\",\"Body\":{\"Fields\":[{\"Name\":\"Heading >>>> Background Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile Wide >>>> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Specialties\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Blog >>>> Post Name\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald >>>> House, St. Luke’s Children’s find new ways to help >>>> families\"},{\"Name\":\"Blog Summary\",\"Type\":\"Rich >>>> Text\",\"Value\":\"\"},{\"Name\":\"Share Summary\",\"Type\":\"Multi-Line >>>> Text\",\"Value\":\"\"},{\"Name\":\"ShareTitle\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >>>> ways to help families\"},{\"Name\":\"Blog Post >>>> Date\",\"Type\":\"Datetime\",\"Value\":\"2021-01-18T10:10:00Z\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Better Together\"},{\"Name\":\"Rss >>>> Link\",\"Type\":\"General >>>> Link\",\"Value\":\"\"},{\"Name\":\"Providers\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Main >>>> Blog Image Caption\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"\"},{\"Name\":\"Procedures and >>>> Treatments\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Special >>>> Services\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Page >>>> Title\",\"Type\":\"Single-Line Text\",\"Value\":\"Ronald McDonald House, >>>> St. Luke’s Children’s finding new ways to help >>>> families\"},{\"Name\":\"ShareImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Share >>>> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Channels\",\"Type\":\"Multilist\",\"Value\":\"Better >>>> Together\"},{\"Name\":\"Blog >>>> Tags\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileHeadline\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >>>> ways to help families\"},{\"Name\":\"Include in >>>> Sitemap\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Facilities\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"TileImage\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Tile >>>> Category\",\"Type\":\"Droptree\",\"Value\":\"Blog >>>> Post\"},{\"Name\":\"BreadcrumbTitle\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Ronald McDonald House, St. Luke’s Children’s find new >>>> ways to help >>>> families\"},{\"Name\":\"Author\",\"Type\":\"Droplink\",\"Value\":\"{E9CF1FC9-EF41-4B6F-9D78-F206A5997A84}\"},{\"Name\":\"Restricted >>>> To Pages\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Meta >>>> Keywords\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"\"},{\"Name\":\"Associated Content >>>> Type\",\"Type\":\"Droptree\",\"Value\":\"Blog Post\"},{\"Name\":\"Main Blog >>>> Image\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"TileSummary\",\"Type\":\"Rich >>>> Text\",\"Value\":\"\"},{\"Name\":\"Meta Description\",\"Type\":\"Multi-Line >>>> Text\",\"Value\":\"\"},{\"Name\":\"Health >>>> Topics\",\"Type\":\"Multilist\",\"Value\":\"\"},{\"Name\":\"Icon\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"NavigationTitle\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"St. Luke’s Blogs\"},{\"Name\":\"Include in Search >>>> Index\",\"Type\":\"Checkbox\",\"Value\":\"1\"},{\"Name\":\"Conditions\",\"Type\":\"Treelist\",\"Value\":\"\"},{\"Name\":\"Heading >>>> Sub Text\",\"Type\":\"Single-Line Text\",\"Value\":\"Highlights from St. >>>> Luke’s and our community partners to improve >>>> health.\"},{\"Name\":\"typeaheadRollupCat\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"\"},{\"Name\":\"BlogPostYear\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"2021\"},{\"Name\":\"AuthorName\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Anna >>>> Fritz\"},{\"Name\":\"BlogCategory\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"News and >>>> Community\"}],\"Modules\":{\"Fields\":[{\"Name\":\"Content\",\"Type\":\"Rich >>>> Text\",\"Value\":\"\"},{\"Name\":\"Image >>>> Position\",\"Type\":\"Droptree\",\"Value\":\"Right\"},{\"Name\":\"Image >>>> Source\",\"Type\":\"Image\",\"Value\":\"\"},{\"Name\":\"Image >>>> Content\",\"Type\":\"Rich Text\",\"Value\":\"<p>For more than three >>>> decades, Ronald McDonald House Charities of Idaho has provided housing to >>>> families with children seeking medical care.</p>\\n<p>It also has found new >>>> ways to help during the COVID-19 era. </p>\\n<p>When the novel coronavirus >>>> gained a foothold in Idaho in March 2020, the organization had to put >>>> safety first and made the tough decision to temporarily stop accepting new >>>> families not already staying at its new Boise house. Instead, the >>>> organization paid for hotel rooms for families it could not >>>> accommodate. </p>\\n<p>“When everything happened, because we had to >>>> pull back services, we were trying to look for other ways we could help >>>> families,” said Taylor Munson, communications manager at Ronald McDonald >>>> House Charities of Idaho. </p>\\n<p>“They are obviously already in a >>>> stressful situation with a sick child, but the pandemic amplified that >>>> because there is even more unknown now.”</p>\\n<p>So, how could their staff >>>> keep serving families with kids in need? </p>\\n<p>The team at RMHCI >>>> decided to start assembling lunch boxes filled with meals for families with >>>> kids at St. Luke’s Children’s Hospital. </p>\\n<p>Since March, the staff >>>> has provided 4,770 meals to families at St. Luke’s.</p>\\n<p>“The lunches >>>> provided by the Ronald McDonald House have been a true blessing for our >>>> families in pediatrics, the pediatric ICU and the newborn ICU,” said Sherry >>>> Iverson, director of patient and family services at St. Luke’s >>>> Children’s.</p>\\n<p>“Being at the bedside of their children of all ages is >>>> top priority for parents and remembering to take care of themselves is >>>> easily forgotten. These lunches carefully assembled by the Ronald McDonald >>>> team and then delivered to their room provide a break, healthy food and a >>>> chance to reenergize during a very stressful time.” </p>\\n<p>An additional >>>> 920 meals have been provided by RMHCI to families with children receiving >>>> care at Saint Alphonsus Health System. </p>\\n<p>The Ronald McDonald staff >>>> provides the lunch boxes four days a week, typically including sandwiches, >>>> fruit and chips, as well as snack bags. The total cost of the lunches so >>>> far has been about $24,000.</p>\\n<p>“Without these wonderful care >>>> packages, many parents would go all day without food,” Iverson said. “This >>>> partnership has been so important during this COVID >>>> pandemic.”</p>\\n<p>Some of the food items are donated from local >>>> organizations, while others are purchased by staff and assembled in the >>>> kitchen at the new Ronald McDonald House facility, near the St. Luke’s >>>> Boise Medical Center. </p>\\n<p>St. Luke’s employees pick up the meals and >>>> take them across the street to the children’s hospital.</p>\\n<p>“The >>>> feedback that we’ve gotten from families and nurses and people over at St. >>>> Luke’s is that it’s so helpful because families either may not have money >>>> to get food or they don’t want to leave their child’s bedside,” Munson >>>> said.</p>\\n<p>The Ronald McDonald House started accepting new families >>>> again at its facility in May 2020. St. Luke’s Children’s Hospital is the >>>> only children’s hospital in Idaho, which has led to a strong partnership >>>> between the medical center and nearby Ronald McDonald House. </p>\\n<p>“It >>>> has been very collaborative with St. Luke’s. We wanted to make sure what we >>>> were going to be doing was beneficial,” Munson said. </p>\\n<p>The program >>>> will continue through the end of March, marking one full year of providing >>>> meals, and then the Ronald McDonald House staff will reevaluate for short- >>>> and long-term plans, Munson said. </p>\\n<p>“The pandemic obviously isn’t >>>> ideal, but it did allow us find new ways of helping families,” Munson said. >>>> “A lot of our focus is family centered care—that’s really our goal, and >>>> feeding families is a big part of >>>> that.”</p>\\n<p><br>\\n</p>\"},{\"Name\":\"Channel\",\"Type\":\"Droptree\",\"Value\":\"Better >>>> Together\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Better Together\"},{\"Name\":\"Number to >>>> Display\",\"Type\":\"Integer\",\"Value\":\"4\"},{\"Name\":\"Related >>>> Item\",\"Type\":\"Droptree\",\"Value\":\"St Lukes Childrens >>>> Hospital\"},{\"Name\":\"Heading\",\"Type\":\"Single-Line >>>> Text\",\"Value\":\"Related >>>> Hospital\"}]}},\"Facets\":[\"Blogs\",\"Article\"],\"Title\":\"Ronald >>>> McDonald House, St. Luke’s Children’s find new ways to help >>>> families\",\"Summary\":\"\"}}" >>>> } >>>> >>>> ________________________________ >>>> From: Mikhail Khludnev <m...@apache.org <mailto:m...@apache.org>> >>>> Sent: Saturday, November 26, 2022 1:28 PM >>>> To: users@solr.apache.org <mailto:users@solr.apache.org> >>>> <users@solr.apache.org <mailto:users@solr.apache.org>> >>>> Subject: Re: Is there a way to run the entire payload of a request through >>>> a charFilter and not just the fields? >>>> >>>> Hi Matthew. Can it be https: //urldefense. com/v3/__https: //solr. apache. >>>> org/guide/solr/latest/configuration-guide/script-update-processor. >>>> html__;!!FkC3_z_N!ON6B9iNNwK7AkdwAKGpLzLAzNKXR4m8SIom95HENXZNK381f6vhLlbAf5l7Z2mpVNUNJWAP2dw$ >>>> ? On Sat, >>>> ZjQcmQRYFpfptBannerStart >>>> This Message Is From an External Sender >>>> This message came from outside the St. Luke's email system. >>>> >>>> ZjQcmQRYFpfptBannerEnd >>>> >>>> Hi Matthew. >>>> Can it be >>>> >>>> https://urldefense.com/v3/__https://solr.apache.org/guide/solr/latest/configuration-guide/script-update-processor.html__;!!FkC3_z_N!ON6B9iNNwK7AkdwAKGpLzLAzNKXR4m8SIom95HENXZNK381f6vhLlbAf5l7Z2mpVNUNJWAP2dw$ >>>> <https://urldefense.com/v3/__https://solr.apache.org/guide/solr/latest/configuration-guide/script-update-processor.html__;!!FkC3_z_N!ON6B9iNNwK7AkdwAKGpLzLAzNKXR4m8SIom95HENXZNK381f6vhLlbAf5l7Z2mpVNUNJWAP2dw$%3E%3E>? >>>> >>>> On Sat, Nov 26, 2022 at 1:15 AM Matthew Castrigno <castr...@slhs.org >>>> <mailto:castr...@slhs.org>> >>>> wrote: >>>> >>>>> I need to filter out some characters in a payload so that SOLR will >>>>> recognize the payload as a JSON document. >>>>> >>>>> The solr.MappingCharFilterFactory functionality is what I need but I need >>>>> to run over the entire payload and not just the fields. >>>>> >>>>> I cannot change the payload prior to submitting to SOLR. >>>>> >>>>> Is there any way to accomplish this? >>>>> >>>>> Any insights are most appreciated. >>>>> >>>>> Thank you. >>>>> >>>>> ---------------------------------------------------------------------- >>>>> "This message is intended for the use of the person or entity to which it >>>>> is addressed and may contain information that is confidential or >>>>> privileged, the disclosure of which is governed by applicable law. If the >>>>> reader of this message is not the intended recipient, you are hereby >>>>> notified that any dissemination, distribution, or copying of this >>>>> information is strictly prohibited. If you have received this message by >>>>> error, please notify us immediately and destroy the related message." >>>>> >>>> >>>> >>>> -- >>>> Sincerely yours >>>> Mikhail Khludnev >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> "This message is intended for the use of the person or entity to which it >>>> is addressed and may contain information that is confidential or >>>> privileged, the disclosure of which is governed by applicable law. If the >>>> reader of this message is not the intended recipient, you are hereby >>>> notified that any dissemination, distribution, or copying of this >>>> information is strictly prohibited. If you have received this message by >>>> error, please notify us immediately and destroy the related message." >>>> >>> >>> >>> -- >>> Sincerely yours >>> Mikhail Khludnev >>> >>> >>> ---------------------------------------------------------------------- >>> "This message is intended for the use of the person or entity to which it >>> is addressed and may contain information that is confidential or >>> privileged, the disclosure of which is governed by applicable law. If the >>> reader of this message is not the intended recipient, you are hereby >>> notified that any dissemination, distribution, or copying of this >>> information is strictly prohibited. If you have received this message by >>> error, please notify us immediately and destroy the related message." >> >> "This message is intended for the use of the person or entity to which it is >> addressed and may contain information that is confidential or privileged, >> the disclosure of which is governed by applicable law. If the reader of this >> message is not the intended recipient, you are hereby notified that any >> dissemination, distribution, or copying of this information is strictly >> prohibited. If you have received this message by error, please notify us >> immediately and destroy the related message." > > > > ---------------------------------------------------------------------- > "This message is intended for the use of the person or entity to which it is > addressed and may contain information that is confidential or privileged, the > disclosure of which is governed by applicable law. If the reader of this > message is not the intended recipient, you are hereby notified that any > dissemination, distribution, or copying of this information is strictly > prohibited. If you have received this message by error, please notify us > immediately and destroy the related message."
