Hello Igniters,
I would like to consult with you the following:
I am trying to understand the performance of text search in Ignite 2.3 on 1
milion of entities with one field: name.
I execute the search and take first 10 entries from the cursor.
I have noticed that the time of getting first entry (or calling first
hasNext()) is equal to the time of getting all entries.
Seems that internally the cursor is prepared by fetching and caching all
query data.
Also the processing time is growing significantly with the number of query
hits. for example when query returns 89242 items (see below), the execution
time of the first hasNext() is 1327 ms. what is rather unacceptable for our
auto-suggest solution (we are planning to search over hundreds of millions
of entries). Example and code is below.
Could you confirm this as a bug or feature?
Regards,
zbyszek
All time: (393 items ) 42 ms.
Cursor time: 1 ms.
Iterator time: 0 ms.
Next time: (0) 45 ms.
Next time: (1) 0 ms.
Next time: (2) 0 ms.
Next time: (3) 0 ms.
Next time: (4) 0 ms.
Next time: (5) 0 ms.
Next time: (6) 0 ms.
Next time: (7) 0 ms.
Next time: (8) 0 ms.
Next time: (9) 0 ms.
Result time: 45 ms.
All time: (2385 items ) 79 ms.
Cursor time: 1 ms.
Iterator time: 0 ms.
Next time: (0) 122 ms.
Next time: (1) 0 ms.
Next time: (2) 0 ms.
Next time: (3) 0 ms.
Next time: (4) 0 ms.
Next time: (5) 0 ms.
Next time: (6) 0 ms.
Next time: (7) 0 ms.
Next time: (8) 0 ms.
Next time: (9) 0 ms.
Result time: 122 ms.
All time: (28026 items ) 485 ms.
Cursor time: 0 ms.
Iterator time: 0 ms.
Next time: (0) 500 ms.
Next time: (1) 0 ms.
Next time: (2) 0 ms.
Next time: (3) 0 ms.
Next time: (4) 0 ms.
Next time: (5) 0 ms.
Next time: (6) 0 ms.
Next time: (7) 0 ms.
Next time: (8) 0 ms.
Next time: (9) 0 ms.
Result time: 500 ms.
All time: (89242 items ) 1383 ms.
Cursor time: 0 ms.
Iterator time: 0 ms.
Next time: (0) 1327 ms.
Next time: (1) 0 ms.
Next time: (2) 0 ms.
Next time: (3) 0 ms.
Next time: (4) 0 ms.
Next time: (5) 0 ms.
Next time: (6) 0 ms.
Next time: (7) 0 ms.
Next time: (8) 0 ms.
Next time: (9) 0 ms.
Result time: 1327 ms.
public QueryEntity createSearchQuery(String entity) {
QueryEntity queryEntity = new
QueryEntity(String.class.getTypeName(), entity);
List<QueryIndex> indexes = new ArrayList<>();
Set<String> fields = new LinkedHashSet<>(Arrays.asList("name"));
fields.forEach(f -> {
queryEntity.addQueryField(f, String.class.getTypeName(), f);
LinkedHashMap<String, Boolean> indexField = new
LinkedHashMap<>();
indexField.put(f, false);
QueryIndex index = new QueryIndex(indexField,
QueryIndexType.FULLTEXT);
index.setName(f + "Index");
indexes.add(index);
});
queryEntity.setIndexes(indexes);
return queryEntity;
}
public SearchResult search(Search search) {
String query = search.getTerm();
TextQuery<String, BinaryObject> textQuery = new
TextQuery<>("Search", query);
textQuery.setLocal(true);
textQuery.setPageSize(10);
IgniteCache<String, BinaryObject> cache = ignite.cache("search");
QueryCursor<Cache.Entry<String, BinaryObject>> cursor =
cache.withKeepBinary().query(textQuery);
long startTime = System.currentTimeMillis();
int size = cursor.getAll().size();
System.out.println("All time: " + "(" + size + " items ) " +
(System.currentTimeMillis() - startTime) + " ms.");
startTime = System.currentTimeMillis();
cursor = cache.withKeepBinary().query(textQuery);
System.out.println("Cursor time: " + (System.currentTimeMillis() -
startTime) + " ms.");
startTime = System.currentTimeMillis();
Iterator<Cache.Entry<String, BinaryObject>> iterator =
cursor.iterator();
System.out.println("Iterator time: " + (System.currentTimeMillis() -
startTime) + " ms.");
List<SearchResult.Entry> res = new ArrayList<>();
int counter = 0;
startTime = System.currentTimeMillis();
while (/*iterator.hasNext() && */ counter < search.getMaxRecords()){
long nextStartTime = System.currentTimeMillis();
Cache.Entry<String, BinaryObject> entry = iterator.next();
System.out.println("Next time: " + "(" + counter + ") " +
(System.currentTimeMillis() - nextStartTime) + " ms.");
String key = entry.getKey();
BinaryObject obj = entry.getValue();
StringBuilder sb = new StringBuilder();
for (String f : fields) {
String v = obj.field(f);
if (v != null &&
v.toLowerCase().contains(search.getTerm().toLowerCase())) {
sb.append(f).append(": ").append(v).append(", ");
}
}
SearchResult.Entry dataRow = new SearchResult.Entry(key,
extractType(key), obj.field("name"), sb.toString());
res.add(dataRow);
counter++;
}
System.out.println("Result time: " + (System.currentTimeMillis() -
startTime) + " ms.");
return new SearchResult(System.currentTimeMillis() - startTime,
search.getTerm(), res.toArray(new SearchResult.Entry[0]));
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
