This is what I am doing with Carmine and it seems to work properly.
First some setup in the repl:
user=>(require '[taoensso.carmine :as car])
nil
user=> (def conn {:pool {} :spec {}})
#'user/conn
user=> (defn raw-str-set [k ^String v] (car/set k (car/raw (.getBytes v
"UTF-8"))))
#'user/raw-str-set
user=> (defn raw-str-get [k] (car/parse (car/get k) #(String. ^bytes %
"UTF-8")))
#'user/raw-str-get
Now I save a test string:
user=> (car/wcar conn (raw-str-set "test" "a\"b"))
"OK"
In a redis-cli terminal:
$ redis-cli
127.0.0.1:6379> get "test"
"a\"b"
127.0.0.1:6379>
The escaped \" you see in redis-cli is just for printing. The actual bytes
stored at the "test" key are 0x61 0x22 0x62
Now lets pull this out of redis with Carmine:
user=> (car/wcar conn (raw-str-get "test"))
"a\"b"
user=> (count *1)
3
As you can see, we round-tripped without any strange extra escaping.
Probably your java app can just use Jedis client.get("test") and
client.set("test","a-string"), since I think it does UTF-8 encode and
decode of strings and no other serialization.
You still need to use JSON on top of this so you can get richer data
structures instead of just strings.
On Thursday, July 9, 2015 at 4:18:51 PM UTC-5, gingers...@gmail.com wrote:
>
> I am sorry, I should have explained this sooner. When I do this:
>
> (.getBytes document-as-string "UTF-8")
>
> The quote marks are always escaped. When I do this:
>
> document-as-byte-array (bytes (byte-array (map (comp byte int)
> document-as-string)))
>
> The quote marks are sometimes escaped, but most of the time they are not.
>
> I need to avoid having those quote marks escaped. I am not clear what is
> causing the escaping.
>
>
>
>
> On Thursday, July 9, 2015 at 12:01:51 PM UTC-4, Francis Avila wrote:
>>
>> your document-as-byte-array is wrong--it only handles ascii, and very
>> inefficiently too.
>>
>> Just say (.getBytes document-as-string "UTF-8") to get a utf-8 encoding
>> of the string as a byte array.
>>
>> On Thu, Jul 9, 2015 at 10:41 AM, <gingers...@gmail.com> wrote:
>>
>>>
>>> > As for the escaped quotes, you may be using pr or prn to print, or
>>> maybe you are
>>> > using pr-str to produce the string representation. I can't be sure.
>>>
>>> At the moment I create the string like this:
>>>
>>> document-as-string (str "{\"transaction-id\" : \""
>>> transaction-id "\", \"message\" : \"" message "\"}")
>>> document-as-byte-array (bytes (byte-array (map (comp byte int)
>>> document-as-string)))
>>>
>>> The crazy thing is that this sometimes works, but other times the quote
>>> marks appear in Redis as escaped quote marks. As near as I can tell, the
>>> important factor is the length of the string. A short string is likely to
>>> have its quote marks escaped. A long string does not have its quote marks
>>> escaped.
>>>
>>> My co-worker is working on the Java app. I am working on the Clojure
>>> app. We can both adjust out apps freely, just so long as we can get data to
>>> and from each other, in a manner that allows us to eventually cast the data
>>> to and from JSON.
>>>
>>> Any suggestions are welcome.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, July 8, 2015 at 8:58:53 PM UTC-4, Francis Avila wrote:
>>>>
>>>> Who is saving these strings, and who is reading them? Do you have
>>>> complete control over both apps, or does one of them need to be aligned
>>>> with the other?
>>>>
>>>> If the Java app is the baseline, you need to know the exact details of
>>>> the format of the data it saves. Just knowing "it's JSON" is not enough,
>>>> because there may be other data types (e.g. dates) that don't have native
>>>> JSON representations. (The go-to JSON de/encoder for clojure is cheshire:
>>>> https://github.com/dakrone/cheshire )
>>>>
>>>> I took a quick look at Jedis and the easy, default way of using it is
>>>> with strings. It will encode strings to UTF-8 before sending to Redis and
>>>> decode from UTF-8 on read. You can set raw byte arrays too (which will not
>>>> be altered in any way before sending), but it's not clear to me how it can
>>>> read out raw byte arrays. (I'm sure there's a way, but it's not
>>>> immediately
>>>> obvious.)
>>>>
>>>> As for the escaped quotes, you may be using pr or prn to print, or
>>>> maybe you are using pr-str to produce the string representation. I can't
>>>> be
>>>> sure.
>>>>
>>>> On Wednesday, July 8, 2015 at 5:31:27 PM UTC-5, gingers...@gmail.com
>>>> wrote:
>>>>>
>>>>> And I have another stupid question. Using the above code, I am
>>>>> sometimes getting strings in Redis that have escaped quotation marks,
>>>>> like
>>>>> this:
>>>>>
>>>>> " \"transaction-id\" : \" 1ec47c2e-21ee-427c-841c-80a0f89f55d7 \"
>>>>> \"debrief\" : \" Susan Hilly at Citi called to get a quotation for
>>>>> discounted weekly car rental for approximately 3 cars per week, or 150
>>>>> rentals annually. \" "
>>>>>
>>>>> Why is that happening?
>>>>>
>>>>>
>>>>>
>>>>> On Wednesday, July 8, 2015 at 5:38:20 PM UTC-4, gingers...@gmail.com
>>>>> wrote:
>>>>>>
>>>>>> Francis Avila,
>>>>>>
>>>>>> Thank you for your response. The Java app is using Jedis and the
>>>>>> Clojure app is using Carmine. I'm wondering if you can suggest what you
>>>>>> think would be the easiest way to allow these 2 apps to understand each
>>>>>> other's strings?
>>>>>>
>>>>>> You were correct about how unsafe the above code was. I tested it for
>>>>>> less than 15 minutes and ran into the fact that a \n newline made a mess
>>>>>> of
>>>>>> everything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wednesday, July 8, 2015 at 5:15:35 PM UTC-4, Francis Avila wrote:
>>>>>>>
>>>>>>> You are running into Carmine's automatic nippy serialization.
>>>>>>> https://github.com/ptaoussanis/carmine#serialization
>>>>>>>
>>>>>>> Redis only stores byte arrays (what it calls "strings"). Carmine
>>>>>>> uses the nippy library (the meaning of "NPY" in your byte stream) to
>>>>>>> represent rich types compactly as bytes.
>>>>>>> https://github.com/ptaoussanis/nippy
>>>>>>>
>>>>>>> If you give Carmine a byte array to store, it will store it directly
>>>>>>> without nippy-encoding it. E.g. (.getBytes "{}" "UTF-8")
>>>>>>>
>>>>>>> BTW your document-as-string example is extremely unsafe: how will
>>>>>>> you reliably read this message out again? e.g. what if the 'debrief'
>>>>>>> string
>>>>>>> contains a single quote? Use a proper serialization format.
>>>>>>>
>>>>>>> So the key is to have both your Clojure and Java app store *bytes*
>>>>>>> in Redis using the same serialization. You can store anything you want
>>>>>>> (nippy, utf-8-encoded json, fressian, bson, utf-8 xml, utf-16 java
>>>>>>> strings,
>>>>>>> whatever) as long as it's bytes and it's read and written the same way
>>>>>>> in
>>>>>>> all your apps.
>>>>>>>
>>>>>>> The Redis library your Java app is using may have its own automatic
>>>>>>> de/serialization, too. You need to find out what it's doing and either
>>>>>>> work
>>>>>>> with this or turn it off, just like with Carmine.
>>>>>>>
>>>>>>> Nippy unfortunately does not have a Java API out of the box:
>>>>>>> https://github.com/ptaoussanis/nippy/issues/66
>>>>>>>
>>>>>>>
>>>>>>> On Wednesday, July 8, 2015 at 3:35:49 PM UTC-5, gingers...@gmail.com
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I am not sure if this is a Clojure question or a Java question. I
>>>>>>>> don't know Java, so I could use whatever help folks can offer.
>>>>>>>>
>>>>>>>> "primitive string" here means what I can write when I am at the
>>>>>>>> terminal.
>>>>>>>>
>>>>>>>> We have 2 apps, one in Clojure, one in Java. They talk to each
>>>>>>>> other via Redis. I know the Java app can read stuff out of Redis,
>>>>>>>> using our
>>>>>>>> "transaction-id", if I use the terminal and open up "redis-clj" and
>>>>>>>> write a
>>>>>>>> string directly from the terminal. But I have this Clojure code, which
>>>>>>>> depends on Peter Taoussanis's Carmine library:
>>>>>>>> (defn worker [document]
>>>>>>>> {:pre [(string? (:transaction-id document))]}
>>>>>>>> (let [transaction-id (:transaction-id document)
>>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>>> redis-connection {:pool {} :spec {:host "127.0.0.1" :port
>>>>>>>> 6379 }}]
>>>>>>>> (timbre/log :trace " message we will send to NLP "
>>>>>>>> document-as-string)
>>>>>>>> (carmine/wcar redis-connection (carmine/set transaction-id
>>>>>>>> document))
>>>>>>>> (loop [document-in-redis (carmine/wcar redis-connection
>>>>>>>> (carmine/get transaction-id))]
>>>>>>>>
>>>>>>>> (if-not (.contains (first document-in-redis) "processed")
>>>>>>>> (recur (carmine/wcar redis-connection (carmine/get
>>>>>>>> transaction-id)))
>>>>>>>> (do
>>>>>>>> (carmine/wcar redis-connection (carmine/del
>>>>>>>> transaction-id))
>>>>>>>> document-in-redis)))))
>>>>>>>>
>>>>>>>> This line in particular, I have tried doing this several ways:
>>>>>>>>
>>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>>>
>>>>>>>> In Redis, I expect to see:
>>>>>>>>
>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>>> responded
>>>>>>>> to our proposal and said his company is read to move forward. The rate
>>>>>>>> of
>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they shred
>>>>>>>> about
>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue
>>>>>>>> annually. I
>>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>>
>>>>>>>> But if I then launch redis-cli, I see:
>>>>>>>>
>>>>>>>>
>>>>>>>> 127.0.0.1:6379> keys *
>>>>>>>> 1) "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>>>
>>>>>>>> 127.0.0.1:6379> get "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>>> "\x00>NPY\b\x00\x00\x01\xfc\xf1\xfe\x1b\x00\x00\x00\nj\nip-addressi\x0e165.254.84.238j\x05tokeni$46b87d64-cff3-4b8b-895c-e089ac59544dj\x0bapi-versioni\x02v1j\x0etransaction-idi$42e574e7-3b80-424a-b9ff-01072f1e0358j\adebrief\r\x00\x00\x01YSmeek
>>>>>>>>
>>>>>>>> Hallie of Withers, Smeg, Harrington and Norvig responded to our
>>>>>>>> proposal
>>>>>>>> and said his company is rea-\x00\xf1\x06move forward. The
>>>>>>>> raty\x00\xf0\x0c$400 per ton of shredded pa\x16\x00\xf1\bwas
>>>>>>>> acceptable to
>>>>>>>> them,q\x00Bthey0\x00\x80 about 2E\x00\x10sF\x00\xf1Ldocuments every
>>>>>>>> month.
>>>>>>>> $96,000 in potential revenue annually. I will meet with him
>>>>>>>> tomorrow{\x00\"we#\x00@sign\x92\x00\xa0 contract."
>>>>>>>>
>>>>>>>>
>>>>>>>> I don't know what all of those extra characters are. The Java app
>>>>>>>> is not picking this item up, so I assume the Java app is not seeing
>>>>>>>> this as
>>>>>>>> a string. I expected this to look the same as if I had written this at
>>>>>>>> the
>>>>>>>> terminal:
>>>>>>>>
>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>>> responded
>>>>>>>> to our proposal and said his company is read to move forward. The rate
>>>>>>>> of
>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they shred
>>>>>>>> about
>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue
>>>>>>>> annually. I
>>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>>
>>>>>>>> I assume it is easy to get a string into a format that can be
>>>>>>>> understood by both a Clojure app and a Java app. I don't care what
>>>>>>>> format
>>>>>>>> that is, but it needs to be consistent.
>>>>>>>>
>>>>>>>> Can anyone make suggestions about what I can do to make sure the
>>>>>>>> Clojure app and the Java app both write to Redis using a format that
>>>>>>>> the
>>>>>>>> other will understand? In particular, both apps need to see the
>>>>>>>> "'transaction-id".
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Clojure" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/clojure/9swfBxAbT90/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> clojure+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
