https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835
--- Comment #68 from Luis Bataller <[email protected]> --- (In reply to David Cook from comment #65) > (In reply to Luis Bataller from comment #63) > > Hi, > > > > Thanks for picking this up — indexing item data in Elasticsearch is > > something > > Koha has needed for a long time. > > > > We went through this same design question some months ago and evaluated both > > options: a separate item index, and items as `nested` objects inside the > > biblio > > document. We ended up choosing `nested`. > > Great to meet you at the dev meeting last night, Luis. > > I agree with your analysis regarding having a separate item index vs having > nested objects. To me, having a separate item index doesn't make sense, > because it would be very inefficient at a large scale. > > That said, what have you observed with bib records with many (e.g. 200-1000 > items) nested inside the JSON document in Elasticsearch? > > I am supportive of having nested items, because it allows for better > searching, but I have only tried it with relatively small numbers of items > embedded in the bib index. > > Actually with Elasticsearch I think that you can do includes/excludes with > source filtering... so you could do a search against a large record but just > retrieve the bib record when showing the search results... that could be > very very useful. (In reply to David Cook from comment #66) > @Luis, I mentioned it a bit in the chat last night, but the reason we > haven't used nested items in the past in Elasticsearch is because we were > trying to keep feature parity between Elasticsearch and Zebra, and Zebra has > a flat index structure. > > However, as the use of Elasticsearch in Koha has matured/evolved, we have > started doing more Elasticsearch-specific changes. And I think that your > idea for nested items would be a really good Elasticsearch-specific change. > > I think we should be taking advantage of Elasticsearch's capabilities to > improve Koha's search here. Hi David, great to meet you at the meeting too. You're right that this approach wouldn't work with Zebra. The changes were made to the ES indexing only, leaving Zebra with the same behaviour as before. I understand that not being able to use nested items with Zebra may be disappointing for Zebra users. Regarding the numbers, here is a table with real indexing time measurements: items total(s) gen gen es_wall KB % only items (gen+es_wall) (total) (only items) ----- ------------- ------- ------------ ------- ------ ------------ 0 0.043 0.030 0.003 0.013 2.0 7.0 % 1 0.075 0.065 0.032 0.010 3.7 42.7 % 10 0.208 0.193 0.145 0.015 12.7 69.7 % 50 0.729 0.707 0.662 0.022 51.4 90.8 % 100 1.327 1.295 1.244 0.032 99.6 93.7 % 250 3.434 3.393 3.297 0.041 245.2 96.0 % 500 7.066 7.004 6.852 0.062 487.8 97.0 % 1000 13.864 13.718 13.267 0.146 1502.4 95.7 % As you can see, indexing time grows a lot with the number of items; but if you look closely, most of the time (over 90% from 50 items on) is spent building the nested items. We are working on a solution to remove that hydration time, bringing it down to the time it takes to hydrate a single item. The idea is to take the already hydrated document from ES and replace only the item or items that have changed, so the cost of indexing a change would no longer depend on how many items the record has. Any comments or suggestions you may have would be much appreciated. -- You are receiving this mail because: You are watching all bug changes. _______________________________________________ Koha-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected] website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
