Matthias J. Sax created KAFKA-4499:
--------------------------------------
Summary: Add "getAllKeys" API for querying windowed KTable stores
Key: KAFKA-4499
URL: https://issues.apache.org/jira/browse/KAFKA-4499
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
Currently, both {{KTable}} and windowed-{{KTable}} stores can be queried via IQ
feature. While {{ReadOnlyKeyValueStore}} (for {{KTable}} stores) provide method
{{all()}} to scan the whole store (ie, returns an iterator over all stored
key-value pairs), there is no similar API for {{ReadOnlyWindowStore}} (for
windowed-{{KTable}} stores).
This limits the usage of a windowed store, because the user needs to know what
keys are stored in order the query it. It would be useful to provide possible
APIs like this (only a rough sketch):
- {{keys()}} returns all keys available in the store (maybe together with
available time ranges)
- {{all(long timeFrom, long timeTo)}} that returns all window for a specific
time range
- {{allLatest()}} that returns the latest window for each key
Because this feature would require to scan multiple segments (ie, RockDB
instances) it would be quite inefficient with current store design. Thus, this
feature also required to redesign the underlying window store itself.
Because this is a major change, a KIP
(https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
is required. The KIP should cover the actual API design as well as the store
refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
