Hi Brian,
thanks for the detailed explaination!
I had a look at an object which constantly fails to load even if retrying:
lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 300 Multiple Choices
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept, Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "6PSreYIOL25KOpNyG0XPe7"
<--- Date: Thu, 31 Jan 2013 10:42:41 GMT
<--- Content-Type: text/plain
<--- Content-Length: 56
<---
<--* Siblings:
<--* 50Uz9nvQWwOUBE6USi2gki
<--* 1JsgLs3CE3k2mWsaCEiPp4
cat: Zugriff nicht möglich: 300 Multiple Choices
(/riak/m/Um|18498012|4|0|18298081)
lftp :~> cat
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
<--- ETag: "50Uz9nvQWwOUBE6USi2gki"
<--- Date: Thu, 31 Jan 2013 10:42:49 GMT
<--- Content-Type: application/octet-stream
<--- Content-Length: 0
<---
lftp :~> cat
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
<--- Date: Thu, 31 Jan 2013 10:43:01 GMT
<--- Content-Type: application/json; charset=UTF-8
<--- Content-Length: 114
<---
{"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}
the object has two siblings, one the deleted empty "tombstone" and one
with the new data. And there's a gap auf 2:30min between both siblings.
There's no immediate write after the deletion. I logged the write
operations and this gap is there also. And the java client constantly
fails to load this object.
This objects are user notifications which are identified by their type
(the key) and can be updated frequently but the user also is allowed to
delete a single notification (which he did in this case).
Is there anything I can do to resolve this? A last write wins just for
this case? If I write a new notification to update the old one I don't
care if there's any deletion tombstone...
Ingo
Am 30.01.2013 22:42, schrieb Brian Roach:
Ingo -
Riak is returning an object with no contents (which ends up being an
empty String passed to Jackson).
Unless you've somehow mangled the data yourself (which sounds unlikely
given the bit about the 404 from the command line; more on that in a
bit) what's happening is that you're encountering a tombstone; an
object that has been deleted via a delete operation but hasn't been
removed yet. This causes an "empty" object to be returned (the
tombstone) and causes Jackson to puke (HTTP will actually return this
as a 404 but if you look there's still a X-Riak-Vclock: header with a
vclock).
Probably the best description of how this works in Riak is a post by
Jon Meredith which can be found here:
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
Unfortunately this is something the Java client doesn't know what to
do with when using the default JSONConverter and your own POJOs. And
it's not as simple as "just return null" because of a case where a
tombstone could actually be a sibling and the client then needs to
resolve the conflict which is the next step in the process. It's
something I'm going to have to think about.
As you've discovered, when the tombstone isn't a sibling simply
retrying will often work because by then the delete has fully
completed and the tombstones have been removed from the Riak nodes.
Is there a reason you're rapidly doing a delete then a store (which
triggers that fetch)?
Thanks,
Brian Roach
On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com> wrote:
Hi Dmitri,
it doesn't happen in my code and it does happen while the riak-client tries
to deserialize a fetched object from riak in a "fetch-before-store" (see the
stack), I also get this error randomly while trying just to fetch an object
from the database.
And if I try to fetch the object from the cmdline I just get a 404. So I
would expect the java-client just returns a null-result for this fetch and
not to throw an exception.
All my objects are stored using the riak-java-client and the
json-serializer.
Ahh, just tested: if I retry it sometimes works, although most of the time
still fails (haven't tried with a sleep so far).
Ingo
Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
Hi Ingo.
It's difficult to diagnose the exact reason without looking at your code.
But that error is a JSON parser error. It gets thrown whenever the code
tries to parse an empty string as a json object.
The general-case solution is to validate your strings or input streams
that you're turning into JSON objects, or to catch an exception when
creating that object and deal with it accordingly.
But again, it's hard to say why it's happening exactly, in your case --
try to determine where in your code that's happening and think of ways
some input or result is empty, and check for that.
Dmitri
On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
<ingo.roc...@bluelionmobile.com <mailto:ingo.roc...@bluelionmobile.com>>
wrote:
Hi,
I wrote a java tool to convert part of our data from a
mysql-database into riak. As this tool is running while our system
is still up, it needs to replay all modifications done in the mysql
database, during these modifications I sometimes get this exception
from the riak client:
com.basho.riak.client.convert.__ConversionException:
java.io.EOFException: No content to map to Object due to end of input
com.basho.riak.client.convert.__ConversionException:
java.io.EOFException: No content to map to Object due to end of input
at
com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
at
com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
at
com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
at
com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
at
com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
at java.lang.Thread.run(Thread.__java:722)
Caused by: java.io.EOFException: No content to map to Object due to
end of input
at
org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
at
org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
at
org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
at
com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
... 7 more
it only happens once every few thousand updates and if I check the
object from the cmdline I only get a 404.
Any ideas what might cause this and how to fix/workaround it?
Ingo
_________________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
>>> qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
