[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15617617#comment-15617617
]
Ishan Chattopadhyaya commented on SOLR-5944:
--------------------------------------------
Just discovered another, more common, problem with reordered DBQs and in-place
updates working together. The earlier discussed problem, of resurrecting a
document, is very similar. So, here's a description of both:
SCENARIO 1:
{code}
Imagine, updates on leader are:
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
DBQ (q="updateable_field:1", version=300)
The same on the replica (forwarded):
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
DBQ (q="updateable_field:1", version=300)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
The expected net effect is that no document is deleted, and id=1 document
exists with updateable_field=2.
Here, the DBQ was reordered. When they are executed on the replica, the
version=200 update cannot be applied since there is no document with
(id=1,prevVersion=100). What is required is a resurrection of the document that
was deleted by the DBQ, so that other stored/indexed fields are not lost.
{code}
SCENARIO 2:
{code}
Imagine, updates on leader are:
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
DBQ (q="id:1", version=300)
The same on the replica (forwarded):
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
DBQ (q="id:1", version=300)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
The expected net effect is that the document with id=1 be deleted. But again,
the DBQ is reordered. When executed on replica, update version=200 cannot be
applied, since the id=1 document has been deleted. What is required is for this
update (version=200) to be dropped silently.
{code}
Scenario 1 is rare, scenario 2 would be more common. At the point when the
inplace update (version=200 in both cases) is applied, the replica has no way
to know if the update requires a resurrection of the document, or requires to
be dropped.
Till now, I hadn't considered scenario 2, but for the rare scenario 1, I
resorted to throwing an error so as to throw the replica in LIR. Clearly, in
view of scenario 2, this looks like a bad idea. Here are two potential
solutions that come to mind:
Solution 1:
{code}
In a replica, while applying an in-place update, if the required prevVersion
update cannot be found in tlog or index (due to these reordered DBQs), then
fetch from the leader an update that contains the full document with the
version (for which the update failed at replica). Downside to this approach is
that unstored/non-dv fields will get dropped (as is the case with regular
atomic updates today).
{code}
Solution 2:
{code}
Ensure that DBQs are never reordered from leader -> replica. One approach can
be SOLR-8148. Another could be to block, on the leader, all updates newer than
a DBQ, that has been sent through a different thread, until the DBQ is
processed on leader and all the replicas, and only then process the other
updates.
{code}
Solution 1 seems easier to implement now than solution 2, but solution 2 (if
implemented correctly) seems cleaner. Any thoughts?
> Support updates of numeric DocValues
> ------------------------------------
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
> Reporter: Ishan Chattopadhyaya
> Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
