manojpec commented on a change in pull request #4352:
URL: https://github.com/apache/hudi/pull/4352#discussion_r793426614
##########
File path:
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileReader.java
##########
@@ -133,21 +138,43 @@ public BloomFilter readBloomFilter() {
@Override
public Set<String> filterRowKeys(Set candidateRowKeys) {
- // Current implementation reads all records and filters them. In certain
cases, it many be better to:
- // 1. Scan a limited subset of keys (min/max range of candidateRowKeys)
- // 2. Lookup keys individually (if the size of candidateRowKeys is much
less than the total keys in file)
try {
- List<Pair<String, R>> allRecords = readAllRecords();
- Set<String> rowKeys = new HashSet<>();
- allRecords.forEach(t -> {
- if (candidateRowKeys.contains(t.getFirst())) {
- rowKeys.add(t.getFirst());
- }
- });
- return rowKeys;
+ return filterKeys(new TreeSet<>(candidateRowKeys));
} catch (IOException e) {
- throw new HoodieIOException("Failed to read row keys from " + path, e);
+ LOG.error("Failed to fetch keys from " + path);
+ }
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Set<String> filterKeys(TreeSet<String> sortedCandidateRowKeys) throws
IOException {
+ return filterRecordsImpl(sortedCandidateRowKeys).keySet();
Review comment:
Fixed the interface and filterKeys as per the above suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
