vinishjail97 commented on PR #20107: URL: https://github.com/apache/hudi/pull/20107#issuecomment-5943444310
Hi @chrevanthreddy, thank you for putting this re-port together, and for the initial vector search PRs. I tested this branch end to end on lakegres with the Hugging Face Wikipedia dataset (`CREATE INDEX ... USING VECTOR`, then `hudi_vector_search`). I ran into a few issues along the way and put the fixes I needed in a small stacked PR on your branch: chrevanthreddy/hudi#3. It has 9 commits, and most of them are a few lines each: - accept `vector_index` as an index type (`CREATE INDEX` failed without it) - record the bootstrap instant in the manifest (the freshness check failed every query right after bootstrap) - clamp the coarse KMeans cells to `vector.num_clusters` - store extended RaBitQ planes MSB-first, which the scorer expects for bits >= 3 - validate metric/bits at DDL with an `l2` default, and require COW plus a record index - handle bits=1 when the dimension is not a multiple of 64 - a new end-to-end `TestHoodieVectorIndexSearch`, so CI covers this flow Please feel free to take, change, or drop any of these. You know this code better than I do. The PR description also lists a few follow-ups that I did not try to fix: incremental updates, partitioned RLI (#20106), chunked centroids, and bits=1 cosine/dot scoring. I would be glad to pair on any of them. Thanks again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
