[
https://issues.apache.org/jira/browse/KAFKA-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221621#comment-16221621
]
Richard Yu edited comment on KAFKA-4499 at 10/27/17 5:04 AM:
-------------------------------------------------------------
I have realized that this particular way is flawed and should be reconsidered.
There is an another approach from my understanding, which utilizes provided
methods in TreeMap:
Thread Cache is used as a general storage area, and would be used to access
specific NamedCaches, with each CachingWindowStore being assigned one name.
ThreadCache has a method defined as all(String namespace) which could return
all keys present in the specified NamedCache. However that does not neccesarily
mean it would be sorted.
However, we could utilize TreeMap's descendingKeySet() method to get all keys
in descending order. This would require the definition of a new method in
NamedCache which calls upon descendingKeySet() but in effect it will allow us a
sorted set from which to operate.
was (Author: yohan123):
I have realized that this particular way is flawed and should be reconsidered.
There is an another approach from my understanding, which utilizes provided
methods in TreeMap:
Thread Cache is used as a general storage area, and would be used to access
specific NamedCaches, with each CachingWindowStore being assigned one name.
ThreadCache has a method defined as all(String namespace) which could return
all keys present in the specified NamedCache. However that does not neccesarily
mean it would be sorted.
However, we could utilize TreeMap's descendingKeySet() method to get all keys
in ascending order. This would require the definition of a new method in
NamedCache which calls upon descendingKeySet() but in effect it will allow us a
sorted set from which to operate.
> Add "getAllKeys" API for querying windowed KTable stores
> --------------------------------------------------------
>
> Key: KAFKA-4499
> URL: https://issues.apache.org/jira/browse/KAFKA-4499
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Labels: needs-kip
> Attachments: 4499-CachingWindowStore-v1.patch
>
>
> Currently, both {{KTable}} and windowed-{{KTable}} stores can be queried via
> IQ feature. While {{ReadOnlyKeyValueStore}} (for {{KTable}} stores) provide
> method {{all()}} to scan the whole store (ie, returns an iterator over all
> stored key-value pairs), there is no similar API for {{ReadOnlyWindowStore}}
> (for windowed-{{KTable}} stores).
> This limits the usage of a windowed store, because the user needs to know
> what keys are stored in order the query it. It would be useful to provide
> possible APIs like this (only a rough sketch):
> - {{keys()}} returns all keys available in the store (maybe together with
> available time ranges)
> - {{all(long timeFrom, long timeTo)}} that returns all window for a specific
> time range
> - {{allLatest()}} that returns the latest window for each key
> Because this feature would require to scan multiple segments (ie, RockDB
> instances) it would be quite inefficient with current store design. Thus,
> this feature also required to redesign the underlying window store itself.
> Because this is a major change, a KIP
> (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
> is required. The KIP should cover the actual API design as well as the store
> refactoring.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
