As I understand things: yes. (I do not know if there is an explicit way to tells ldbm that a memory block is no more needed, in which case we could do something about it ... )
On Fri, Jan 15, 2021 at 12:11 PM thierry bordaz <tbor...@redhat.com> wrote: > > > On 1/15/21 11:53 AM, Pierre Rogier wrote: > > Hi Thierry, > > I was rather thinking about the key and value duplication when querying > the DB: > > When using bdb functions that is done implicitly. > bdb either copies the values in the DBT buffer or it alloc/realloc it > When mimicking bdb behavior with LDBM we will have to do that explicitly > in the LDBM plugin: > LDMB returns a memory mapped address that may be ummapped > once the txn is ended. So we must copy the result before closing the > txn. > If we have a read txn that protects the full operation lifespan, then we > could directly use the mapped address without needing to duplicate them. > > ah okay ! nice. > Just a question, if we have a txn covering a search with large candidate > list (unindexed), does that mean that by default each db key/value will > remain mapped until txn commit ? > > thanks > thierry > > > Pierre > > On Fri, Jan 15, 2021 at 10:53 AM thierry bordaz <tbor...@redhat.com> > wrote: > >> >> >> On 1/14/21 12:32 PM, Pierre Rogier wrote: >> >> Hi William, >> >> > It's a scenario we will need to fix via your BE work because of the >> MVCC transaction model that >> > LMDB will force us to adopt :) >> >> As I see things in the early phases the lmdb read txn will probably only >> be managed at the db plugin level rather than at backend level. That >> means that we will have the same inconsistency risk than today (i.e as if >> using bdb and the implicit txn). >> The txn model redesign you are speaking about should only occur in one of >> the last phases (once bdb does no more coexists with lmdb). >> It must be done because it could provide a serious performance boost for >> read operations (IMHO, In most cases we could avoid to duplicate the db >> data) >> >> Pierre, what duplicate are you thinking of ? str2entry ? >> >> But we should not do it while bdb is still around because of the risk of >> lock issue and excessive retries. >> >> Note I put a phasing section in >> >> https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing >> explaining that. But I guess I should move it within Ludwig's document >> that englobs it. >> >> >> Pierre >> >> On Thu, Jan 14, 2021 at 12:01 AM William Brown <wbr...@suse.de> wrote: >> >>> >>> >>> > On 13 Jan 2021, at 21:24, Pierre Rogier <prog...@redhat.com> wrote: >>> > >>> > Thank you Willian, >>> > So far your scenario (entry found when reading base entry but no more >>> existing when computing the candidates) is the only one that matches the >>> symptoms. >>> >>> It's a scenario we will need to fix via your BE work because of the MVCC >>> transaction model that LMDB will force us to adopt :) >>> >>> > And that triggered a thought: >>> > We cannot do anything for SUBTREE and ONE_LEVEL searches >>> > because the fact that the base entry id is not in the candidate may >>> be normal >>> > but IMHO we should improve the BASE search case. >>> > In this case the candidate list is directly set to the base entry id >>> > ==> if the candidate entry (in ldbm_back_next_search_entry) is not >>> found and the scope is BASE then we should return a LDAP_NO_SUCH_ENTRY >>> error .. >>> >>> I suspect that Mark has seen this email and submitted a PR to resolve >>> this exact case :) >>> >>> >>> > >>> > Pierre >>> > >>> > >>> > On Wed, Jan 13, 2021 at 1:45 AM William Brown <wbr...@suse.de> wrote: >>> > Hey there, >>> > >>> > https://github.com/389ds/389-ds-base/pull/4525/files >>> > >>> > I had a look and I can see a few possible contributing factors, but >>> without a core and the exact state I can't be sure if this is correct. It's >>> all just hypothetical from reading the code. >>> > >>> > >>> > The crash is in deref_do_deref_attr() which is called as part of >>> deref_pre_entry(). This is the SLAPI_PLUGIN_PRE_ENTRY_FN which is called by >>> "./ldap/servers/slapd/result.c:1488: rc = plugin_call_plugins(pb, >>> SLAPI_PLUGIN_PRE_ENTRY_FN);" >>> > >>> > >>> > I think what's important here is that the search is conducted in >>> ./ldap/servers/slapd/opshared.c:818 rc = (*be->be_search)(pb); Is *not* >>> in a transaction. That means that while the single search in be_search() is >>> consistent due to an implied transaction, the subsequent search in >>> deref_pre_entry() is likely conducted in a seperate transaction. This >>> allows for other operations to potentially interleave and cause changes - >>> modrdn or delete would certainly be candidates to cause a DN to be remove >>> between these two points. It would be extremely hard to reproduce as a race >>> condition of course. >>> > >>> > >>> > A question you asked is why don't we get a "no such entry" error or >>> similar? I think that this is because build_candidate_list in ldbm_search.c >>> doesn't actually create an error if the base_candidates list is empty, >>> because an IDL is allocated with a value of 0 (no matching entries). this >>> allows the search to proceed, and there are no errors, and the result set >>> is set to NULL with size 0. I can't see where LDAP_NO_SUCH_OBJECT is set in >>> this process, but without looking further into it, my suspicion is that >>> entries of size 0 WONT return an error condition to internal_search_pb, so >>> it's valid for this to be empty. >>> > >>> > Anyway, again, this is just reading the code for 20 minutes, and is >>> not a complete in depth investigation, but maybe it's some ideas about what >>> happened? >>> > >>> > Hope it helps :) >>> > >>> > >>> > >>> > — >>> > Sincerely, >>> > >>> > William Brown >>> > >>> > Senior Software Engineer, 389 Directory Server >>> > SUSE Labs, Australia >>> > _______________________________________________ >>> > 389-devel mailing list -- 389-de...@lists.fedoraproject.org >>> > To unsubscribe send an email to >>> 389-devel-le...@lists.fedoraproject.org >>> > Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> > List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> > List Archives: >>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org >>> > >>> > >>> > -- >>> > -- >>> > >>> > 389 Directory Server Development Team >>> > _______________________________________________ >>> > 389-devel mailing list -- 389-de...@lists.fedoraproject.org >>> > To unsubscribe send an email to >>> 389-devel-le...@lists.fedoraproject.org >>> > Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> > List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> > List Archives: >>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org >>> >>> — >>> Sincerely, >>> >>> William Brown >>> >>> Senior Software Engineer, 389 Directory Server >>> SUSE Labs, Australia >>> _______________________________________________ >>> 389-devel mailing list -- 389-de...@lists.fedoraproject.org >>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org >>> >> >> >> -- >> -- >> >> 389 Directory Server Development Team >> >> _______________________________________________ >> 389-devel mailing list -- 389-de...@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org >> >> >> > > -- > -- > > 389 Directory Server Development Team > > > -- -- 389 Directory Server Development Team
_______________________________________________ 389-devel mailing list -- 389-de...@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-de...@lists.fedoraproject.org
