I have not yet found a solution for this problem. If I do this: (.getBytes v "UTF-8")
Then in Redis the quotes are escaped 100% of the time. What is the standard way to handle this? I do see this on StackOverflow: http://stackoverflow.com/questions/12423071/how-to-remove-escape-characters-from-a-string-in-java which suggests: String noSlashes = input.replace("\\", ""); Would this be considered a hack, or would this be the normal way to do this? On Friday, July 10, 2015 at 6:04:01 PM UTC-4, gingers...@gmail.com wrote: > > > Hmm, well, I am grateful to you for running such a detailed test, and I > should have tested this myself: > > >The actual bytes stored at the "test" key are 0x61 0x22 0x62 > > However, something in our code fails when the quote marks are escaped, but > everything works the way we expect when the quote marks do not appear to be > escaped. I amazed by this: > > user=> (count *1) > 3 > > Perhaps the error is something entirely different from what I was > assuming. > > > > > > On Thursday, July 9, 2015 at 7:52:54 PM UTC-4, Francis Avila wrote: >> >> This is what I am doing with Carmine and it seems to work properly. >> >> First some setup in the repl: >> >> user=>(require '[taoensso.carmine :as car]) >> nil >> user=> (def conn {:pool {} :spec {}}) >> #'user/conn >> user=> (defn raw-str-set [k ^String v] (car/set k (car/raw (.getBytes v >> "UTF-8")))) >> #'user/raw-str-set >> user=> (defn raw-str-get [k] (car/parse (car/get k) #(String. ^bytes % >> "UTF-8"))) >> #'user/raw-str-get >> >> >> Now I save a test string: >> >> user=> (car/wcar conn (raw-str-set "test" "a\"b")) >> "OK" >> >> In a redis-cli terminal: >> >> $ redis-cli >> 127.0.0.1:6379> get "test" >> "a\"b" >> 127.0.0.1:6379> >> >> The escaped \" you see in redis-cli is just for printing. The actual >> bytes stored at the "test" key are 0x61 0x22 0x62 >> >> Now lets pull this out of redis with Carmine: >> >> >> user=> (car/wcar conn (raw-str-get "test")) >> "a\"b" >> user=> (count *1) >> 3 >> >> As you can see, we round-tripped without any strange extra escaping. >> >> Probably your java app can just use Jedis client.get("test") and >> client.set("test","a-string"), since I think it does UTF-8 encode and >> decode of strings and no other serialization. >> >> >> You still need to use JSON on top of this so you can get richer data >> structures instead of just strings. >> >> On Thursday, July 9, 2015 at 4:18:51 PM UTC-5, gingers...@gmail.com >> wrote: >>> >>> I am sorry, I should have explained this sooner. When I do this: >>> >>> (.getBytes document-as-string "UTF-8") >>> >>> The quote marks are always escaped. When I do this: >>> >>> document-as-byte-array (bytes (byte-array (map (comp byte int) >>> document-as-string))) >>> >>> The quote marks are sometimes escaped, but most of the time they are >>> not. >>> >>> I need to avoid having those quote marks escaped. I am not clear what is >>> causing the escaping. >>> >>> >>> >>> >>> On Thursday, July 9, 2015 at 12:01:51 PM UTC-4, Francis Avila wrote: >>>> >>>> your document-as-byte-array is wrong--it only handles ascii, and very >>>> inefficiently too. >>>> >>>> Just say (.getBytes document-as-string "UTF-8") to get a utf-8 encoding >>>> of the string as a byte array. >>>> >>>> On Thu, Jul 9, 2015 at 10:41 AM, <gingers...@gmail.com> wrote: >>>> >>>>> >>>>> > As for the escaped quotes, you may be using pr or prn to print, or >>>>> maybe you are >>>>> > using pr-str to produce the string representation. I can't be sure. >>>>> >>>>> At the moment I create the string like this: >>>>> >>>>> document-as-string (str "{\"transaction-id\" : \"" >>>>> transaction-id "\", \"message\" : \"" message "\"}") >>>>> document-as-byte-array (bytes (byte-array (map (comp byte int) >>>>> document-as-string))) >>>>> >>>>> The crazy thing is that this sometimes works, but other times the >>>>> quote marks appear in Redis as escaped quote marks. As near as I can >>>>> tell, >>>>> the important factor is the length of the string. A short string is >>>>> likely >>>>> to have its quote marks escaped. A long string does not have its quote >>>>> marks escaped. >>>>> >>>>> My co-worker is working on the Java app. I am working on the Clojure >>>>> app. We can both adjust out apps freely, just so long as we can get data >>>>> to >>>>> and from each other, in a manner that allows us to eventually cast the >>>>> data >>>>> to and from JSON. >>>>> >>>>> Any suggestions are welcome. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wednesday, July 8, 2015 at 8:58:53 PM UTC-4, Francis Avila wrote: >>>>>> >>>>>> Who is saving these strings, and who is reading them? Do you have >>>>>> complete control over both apps, or does one of them need to be aligned >>>>>> with the other? >>>>>> >>>>>> If the Java app is the baseline, you need to know the exact details >>>>>> of the format of the data it saves. Just knowing "it's JSON" is not >>>>>> enough, >>>>>> because there may be other data types (e.g. dates) that don't have >>>>>> native >>>>>> JSON representations. (The go-to JSON de/encoder for clojure is >>>>>> cheshire: >>>>>> https://github.com/dakrone/cheshire ) >>>>>> >>>>>> I took a quick look at Jedis and the easy, default way of using it is >>>>>> with strings. It will encode strings to UTF-8 before sending to Redis >>>>>> and >>>>>> decode from UTF-8 on read. You can set raw byte arrays too (which will >>>>>> not >>>>>> be altered in any way before sending), but it's not clear to me how it >>>>>> can >>>>>> read out raw byte arrays. (I'm sure there's a way, but it's not >>>>>> immediately >>>>>> obvious.) >>>>>> >>>>>> As for the escaped quotes, you may be using pr or prn to print, or >>>>>> maybe you are using pr-str to produce the string representation. I can't >>>>>> be >>>>>> sure. >>>>>> >>>>>> On Wednesday, July 8, 2015 at 5:31:27 PM UTC-5, gingers...@gmail.com >>>>>> wrote: >>>>>>> >>>>>>> And I have another stupid question. Using the above code, I am >>>>>>> sometimes getting strings in Redis that have escaped quotation marks, >>>>>>> like >>>>>>> this: >>>>>>> >>>>>>> " \"transaction-id\" : \" 1ec47c2e-21ee-427c-841c-80a0f89f55d7 \" >>>>>>> \"debrief\" : \" Susan Hilly at Citi called to get a quotation for >>>>>>> discounted weekly car rental for approximately 3 cars per week, or 150 >>>>>>> rentals annually. \" " >>>>>>> >>>>>>> Why is that happening? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wednesday, July 8, 2015 at 5:38:20 PM UTC-4, gingers...@gmail.com >>>>>>> wrote: >>>>>>>> >>>>>>>> Francis Avila, >>>>>>>> >>>>>>>> Thank you for your response. The Java app is using Jedis and the >>>>>>>> Clojure app is using Carmine. I'm wondering if you can suggest what >>>>>>>> you >>>>>>>> think would be the easiest way to allow these 2 apps to understand >>>>>>>> each >>>>>>>> other's strings? >>>>>>>> >>>>>>>> You were correct about how unsafe the above code was. I tested it >>>>>>>> for less than 15 minutes and ran into the fact that a \n newline made >>>>>>>> a >>>>>>>> mess of everything. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wednesday, July 8, 2015 at 5:15:35 PM UTC-4, Francis Avila wrote: >>>>>>>>> >>>>>>>>> You are running into Carmine's automatic nippy serialization. >>>>>>>>> https://github.com/ptaoussanis/carmine#serialization >>>>>>>>> >>>>>>>>> Redis only stores byte arrays (what it calls "strings"). Carmine >>>>>>>>> uses the nippy library (the meaning of "NPY" in your byte stream) to >>>>>>>>> represent rich types compactly as bytes. >>>>>>>>> https://github.com/ptaoussanis/nippy >>>>>>>>> >>>>>>>>> If you give Carmine a byte array to store, it will store it >>>>>>>>> directly without nippy-encoding it. E.g. (.getBytes "{}" "UTF-8") >>>>>>>>> >>>>>>>>> BTW your document-as-string example is extremely unsafe: how will >>>>>>>>> you reliably read this message out again? e.g. what if the 'debrief' >>>>>>>>> string >>>>>>>>> contains a single quote? Use a proper serialization format. >>>>>>>>> >>>>>>>>> So the key is to have both your Clojure and Java app store *bytes* >>>>>>>>> in Redis using the same serialization. You can store anything you >>>>>>>>> want >>>>>>>>> (nippy, utf-8-encoded json, fressian, bson, utf-8 xml, utf-16 java >>>>>>>>> strings, >>>>>>>>> whatever) as long as it's bytes and it's read and written the same >>>>>>>>> way in >>>>>>>>> all your apps. >>>>>>>>> >>>>>>>>> The Redis library your Java app is using may have its own >>>>>>>>> automatic de/serialization, too. You need to find out what it's doing >>>>>>>>> and >>>>>>>>> either work with this or turn it off, just like with Carmine. >>>>>>>>> >>>>>>>>> Nippy unfortunately does not have a Java API out of the box: >>>>>>>>> https://github.com/ptaoussanis/nippy/issues/66 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wednesday, July 8, 2015 at 3:35:49 PM UTC-5, >>>>>>>>> gingers...@gmail.com wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am not sure if this is a Clojure question or a Java question. I >>>>>>>>>> don't know Java, so I could use whatever help folks can offer. >>>>>>>>>> >>>>>>>>>> "primitive string" here means what I can write when I am at the >>>>>>>>>> terminal. >>>>>>>>>> >>>>>>>>>> We have 2 apps, one in Clojure, one in Java. They talk to each >>>>>>>>>> other via Redis. I know the Java app can read stuff out of Redis, >>>>>>>>>> using our >>>>>>>>>> "transaction-id", if I use the terminal and open up "redis-clj" and >>>>>>>>>> write a >>>>>>>>>> string directly from the terminal. But I have this Clojure code, >>>>>>>>>> which >>>>>>>>>> depends on Peter Taoussanis's Carmine library: >>>>>>>>>> (defn worker [document] >>>>>>>>>> {:pre [(string? (:transaction-id document))]} >>>>>>>>>> (let [transaction-id (:transaction-id document) >>>>>>>>>> document-as-string (str "{'transaction-id' : '" >>>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" ) >>>>>>>>>> redis-connection {:pool {} :spec {:host "127.0.0.1" :port >>>>>>>>>> 6379 }}] >>>>>>>>>> (timbre/log :trace " message we will send to NLP " >>>>>>>>>> document-as-string) >>>>>>>>>> (carmine/wcar redis-connection (carmine/set transaction-id >>>>>>>>>> document)) >>>>>>>>>> (loop [document-in-redis (carmine/wcar redis-connection >>>>>>>>>> (carmine/get transaction-id))] >>>>>>>>>> >>>>>>>>>> (if-not (.contains (first document-in-redis) "processed") >>>>>>>>>> (recur (carmine/wcar redis-connection (carmine/get >>>>>>>>>> transaction-id))) >>>>>>>>>> (do >>>>>>>>>> (carmine/wcar redis-connection (carmine/del >>>>>>>>>> transaction-id)) >>>>>>>>>> document-in-redis))))) >>>>>>>>>> >>>>>>>>>> This line in particular, I have tried doing this several ways: >>>>>>>>>> >>>>>>>>>> document-as-string (str "{'transaction-id' : '" >>>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" ) >>>>>>>>>> >>>>>>>>>> In Redis, I expect to see: >>>>>>>>>> >>>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358', >>>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig >>>>>>>>>> responded >>>>>>>>>> to our proposal and said his company is read to move forward. The >>>>>>>>>> rate of >>>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they >>>>>>>>>> shred about >>>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue >>>>>>>>>> annually. I >>>>>>>>>> will meet with him tomorrow and we will sign the contract.'} >>>>>>>>>> >>>>>>>>>> But if I then launch redis-cli, I see: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 127.0.0.1:6379> keys * >>>>>>>>>> 1) "42e574e7-3b80-424a-b9ff-01072f1e0358" >>>>>>>>>> >>>>>>>>>> 127.0.0.1:6379> get "42e574e7-3b80-424a-b9ff-01072f1e0358" >>>>>>>>>> "\x00>NPY\b\x00\x00\x01\xfc\xf1\xfe\x1b\x00\x00\x00\nj\nip-addressi\x0e165.254.84.238j\x05tokeni$46b87d64-cff3-4b8b-895c-e089ac59544dj\x0bapi-versioni\x02v1j\x0etransaction-idi$42e574e7-3b80-424a-b9ff-01072f1e0358j\adebrief\r\x00\x00\x01YSmeek >>>>>>>>>> >>>>>>>>>> Hallie of Withers, Smeg, Harrington and Norvig responded to our >>>>>>>>>> proposal >>>>>>>>>> and said his company is rea-\x00\xf1\x06move forward. The >>>>>>>>>> raty\x00\xf0\x0c$400 per ton of shredded pa\x16\x00\xf1\bwas >>>>>>>>>> acceptable to >>>>>>>>>> them,q\x00Bthey0\x00\x80 about 2E\x00\x10sF\x00\xf1Ldocuments every >>>>>>>>>> month. >>>>>>>>>> $96,000 in potential revenue annually. I will meet with him >>>>>>>>>> tomorrow{\x00\"we#\x00@sign\x92\x00\xa0 contract." >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I don't know what all of those extra characters are. The Java app >>>>>>>>>> is not picking this item up, so I assume the Java app is not seeing >>>>>>>>>> this as >>>>>>>>>> a string. I expected this to look the same as if I had written this >>>>>>>>>> at the >>>>>>>>>> terminal: >>>>>>>>>> >>>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358', >>>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig >>>>>>>>>> responded >>>>>>>>>> to our proposal and said his company is read to move forward. The >>>>>>>>>> rate of >>>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they >>>>>>>>>> shred about >>>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue >>>>>>>>>> annually. I >>>>>>>>>> will meet with him tomorrow and we will sign the contract.'} >>>>>>>>>> >>>>>>>>>> I assume it is easy to get a string into a format that can be >>>>>>>>>> understood by both a Clojure app and a Java app. I don't care what >>>>>>>>>> format >>>>>>>>>> that is, but it needs to be consistent. >>>>>>>>>> >>>>>>>>>> Can anyone make suggestions about what I can do to make sure the >>>>>>>>>> Clojure app and the Java app both write to Redis using a format that >>>>>>>>>> the >>>>>>>>>> other will understand? In particular, both apps need to see the >>>>>>>>>> "'transaction-id". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clo...@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+u...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Clojure" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/clojure/9swfBxAbT90/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> clojure+u...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
