Hi Brian,
thanks for fixing this so fast!
Ingo
Am 02.02.2013 19:43, schrieb Brian Roach:
Ingo -
As an FYI, once I started looking into this I found you had stumbled
into quite the can of worms. I just finished a pretty comprehensive
set of changes that will now allow proper handling of tombstones in
the Java client - https://github.com/basho/riak-java-client/pull/195
Sorry you had to be the one, but thanks for letting us know about this!
- Roach
On Thu, Jan 31, 2013 at 10:36 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com> wrote:
Hi Brian,
thanks for the suggestion but I already chose a different solution for now,
if these messages get deleted I just delete the links to the message and
mark the message as "abandoned" and available for reuse. So I don't run into
the conflict if I need to store the message again.
I just started the replay again and let it run for a while to see if this
works for me.
Thanks!
Ingo
Am 31.01.2013 16:54, schrieb Brian Roach:
Ingo -
Unfortunately once you've got a sibling tombstone, things get a bit
tricky. It's not going away until you resolve them which when using
the JSONConverter in the Java client, you can't. Oddly enough, this is
the first time anyone has hit this.
I've got a couple ideas on how to address this properly but I need to
look at some things first.
In the meantime, what I'd suggest as a workaround is to copy and paste
the source for the JSONConverter into your own Converter<T> that
you'll pass to the StoreObject and modify it to return null:
https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141
Have it check to see if riakObject.getValue() returns null and if it
does ... return null. You'll also need to modify your ConflictResolver
to check for null as it iterates through the list of your POJOs that
gets passed to it and act accordingly. If there's only a tombstone,
just return null ... which means you will also need to modify your
Mutation<T> to handle a null being passed to it in the case of there
only being a tombstone.
In the end this may well be what I do but I think I have a slightly
more elegant solution that I want to look into.
I've got an errand I need to run this morning, but I'll get to work on
this as soon as I get back.
Thanks, and sorry for the trouble.
- Brian Roach
On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com> wrote:
Hi Brian,
thanks for the detailed explaination!
I had a look at an object which constantly fails to load even if
retrying:
lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 300 Multiple Choices
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept, Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "6PSreYIOL25KOpNyG0XPe7"
<--- Date: Thu, 31 Jan 2013 10:42:41 GMT
<--- Content-Type: text/plain
<--- Content-Length: 56
<---
<--* Siblings:
<--* 50Uz9nvQWwOUBE6USi2gki
<--* 1JsgLs3CE3k2mWsaCEiPp4
cat: Zugriff nicht möglich: 300 Multiple Choices
(/riak/m/Um|18498012|4|0|18298081)
lftp :~> cat
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
<--- ETag: "50Uz9nvQWwOUBE6USi2gki"
<--- Date: Thu, 31 Jan 2013 10:42:49 GMT
<--- Content-Type: application/octet-stream
<--- Content-Length: 0
<---
lftp :~> cat
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
<--- Date: Thu, 31 Jan 2013 10:43:01 GMT
<--- Content-Type: application/json; charset=UTF-8
<--- Content-Length: 114
<---
{"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}
the object has two siblings, one the deleted empty "tombstone" and one
with
the new data. And there's a gap auf 2:30min between both siblings.
There's
no immediate write after the deletion. I logged the write operations and
this gap is there also. And the java client constantly fails to load this
object.
This objects are user notifications which are identified by their type
(the
key) and can be updated frequently but the user also is allowed to delete
a
single notification (which he did in this case).
Is there anything I can do to resolve this? A last write wins just for
this
case? If I write a new notification to update the old one I don't care if
there's any deletion tombstone...
Ingo
Am 30.01.2013 22:42, schrieb Brian Roach:
Ingo -
Riak is returning an object with no contents (which ends up being an
empty String passed to Jackson).
Unless you've somehow mangled the data yourself (which sounds unlikely
given the bit about the 404 from the command line; more on that in a
bit) what's happening is that you're encountering a tombstone; an
object that has been deleted via a delete operation but hasn't been
removed yet. This causes an "empty" object to be returned (the
tombstone) and causes Jackson to puke (HTTP will actually return this
as a 404 but if you look there's still a X-Riak-Vclock: header with a
vclock).
Probably the best description of how this works in Riak is a post by
Jon Meredith which can be found here:
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
Unfortunately this is something the Java client doesn't know what to
do with when using the default JSONConverter and your own POJOs. And
it's not as simple as "just return null" because of a case where a
tombstone could actually be a sibling and the client then needs to
resolve the conflict which is the next step in the process. It's
something I'm going to have to think about.
As you've discovered, when the tombstone isn't a sibling simply
retrying will often work because by then the delete has fully
completed and the tombstones have been removed from the Riak nodes.
Is there a reason you're rapidly doing a delete then a store (which
triggers that fetch)?
Thanks,
Brian Roach
On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com> wrote:
Hi Dmitri,
it doesn't happen in my code and it does happen while the riak-client
tries
to deserialize a fetched object from riak in a "fetch-before-store"
(see
the
stack), I also get this error randomly while trying just to fetch an
object
from the database.
And if I try to fetch the object from the cmdline I just get a 404. So
I
would expect the java-client just returns a null-result for this fetch
and
not to throw an exception.
All my objects are stored using the riak-java-client and the
json-serializer.
Ahh, just tested: if I retry it sometimes works, although most of the
time
still fails (haven't tried with a sleep so far).
Ingo
Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
Hi Ingo.
It's difficult to diagnose the exact reason without looking at your
code.
But that error is a JSON parser error. It gets thrown whenever the
code
tries to parse an empty string as a json object.
The general-case solution is to validate your strings or input streams
that you're turning into JSON objects, or to catch an exception when
creating that object and deal with it accordingly.
But again, it's hard to say why it's happening exactly, in your case
--
try to determine where in your code that's happening and think of ways
some input or result is empty, and check for that.
Dmitri
On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com
<mailto:ingo.roc...@bluelionmobile.com>>
wrote:
Hi,
I wrote a java tool to convert part of our data from a
mysql-database into riak. As this tool is running while our
system
is still up, it needs to replay all modifications done in the
mysql
database, during these modifications I sometimes get this
exception
from the riak client:
com.basho.riak.client.convert.__ConversionException:
java.io.EOFException: No content to map to Object due to end of
input
com.basho.riak.client.convert.__ConversionException:
java.io.EOFException: No content to map to Object due to end of
input
at
com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
at
com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
at
com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
at
com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
at java.lang.Thread.run(Thread.__java:722)
Caused by: java.io.EOFException: No content to map to Object due
to
end of input
at
org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
at
org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
at
org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
at
com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
... 7 more
it only happens once every few thousand updates and if I check
the
object from the cmdline I only get a 404.
Any ideas what might cause this and how to fix/workaround it?
Ingo
_________________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
>>> qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
