[
https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122147#comment-15122147
]
ASF GitHub Bot commented on KAFKA-3060:
---------------------------------------
GitHub user guozhangwang opened a pull request:
https://github.com/apache/kafka/pull/826
KAFKA-3060: Refactor MeteredStore and RockDBStore Impl
Changes include:
1) Move logging logic from MeteredXXXStore to internal stores, and leave
WindowedStore API clean by removed all internalPut/Get functions.
2) Wrap common logging behavior of InMemory and LRUCache stores into one
class.
3) Fix a bug for StoreChangeLogger where byte arrays are not comparable in
HashSet by using a specified RawStoreChangeLogger.
4) Add a caching layer on top of RocksDBStore with object caching, it
relies on the object's equals and hashCode function to be consistent with
serdes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guozhangwang/kafka K3060
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/826.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #826
----
commit bf4c4cb3dbb5b4066d9c3e0ada5b7ffd98eb129a
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-14T20:27:58Z
add internal source topic for tracking
commit 1485dff08a76c6ff685b0fe72226ce3b629b1d3c
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-14T22:32:08Z
minor fix for this.interSourceTopics
commit 60cafd0885c41f93e408f8d89880187ddec789a1
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-15T01:09:00Z
add KStream windowed aggregation
commit 983a626008d987828deabe45d75e26e909032843
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-15T01:34:56Z
merge from apache trunk
commit 57051720de4238feb4dc3c505053096042a87d9c
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-15T21:38:53Z
v1
commit 4a49205fcab3a05ed1fd05a34c7a9a92794b992d
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-15T22:07:17Z
minor fix on HoppingWindows
commit 9b4127e91c3a551fb655155d9b8e0df50132d0b7
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-15T22:43:14Z
fix HoppingWindows
commit 9649fe5c8a9b2e900e7746ae7b8745bb65694583
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-16T19:00:54Z
add retainDuplicate option in RocksDBWindowStore
commit 8a9ea02ac3f9962416defa79d16069431063eac0
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-16T19:06:12Z
minor fixes
commit 4123528cf4695b05235789ebfca3a63e8a832ffa
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-18T17:55:02Z
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into
K3104
commit 46e8c8d285c0afae6da9ec7437d082060599f3f1
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-18T19:15:47Z
add wordcount and pipe jobs
commit 582d3ac24bfe08edb1c567461971cd35c1f75a00
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-18T21:53:21Z
merge from trunk
commit 5a002fadfcf760627274ddaa016deeaed5a3199f
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-19T00:06:34Z
1. WallClockTimestampExtractor as default; 2. remove windowMS config; 3.
override state dir with jobId prefix;
commit 7425673e523c42806b29a364564a747443712a53
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-19T01:26:11Z
Add PageViewJob
commit ca04ba8d18674c521ad67872562a7671cb0e2c0d
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-19T06:23:05Z
minor changes on topic names
commit 563cc546b3a0dd16d586d2df33c37d2c5a5bfb18
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-19T21:30:11Z
change config importance levels
commit 4218904505363e61bb4c6b60dc5b13badfd39697
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-21T00:11:34Z
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into
K3066
commit 26fb5f3f5a8c9b304c5b1e61778c6bc1d9d5fccb
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-21T06:43:04Z
demo examples v1
commit 6d92a55d770e058183daabb7aaef7675335fbbad
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T00:41:12Z
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into
K3066
commit 929e405058eb61d38510120f3f3ed50cd0cfaf47
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T01:02:04Z
add RollingRocksDBStore
commit 324eb584b97ed3c228347d108d697d2f5133ea99
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T01:23:32Z
modify MeteredWindowStore
commit 7ba2d90fe1de1ca776cea23ff1c2e8f8b3a6c3f2
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T01:35:10Z
remove getter
commit a4d78bac9d84dfd1c7dab4ae465b9115ddc451b3
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T01:36:51Z
remove RollingRocksDB
commit d0e8198ac6a25315d7ab8d21894acf0077f88fde
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T17:24:32Z
adding cache layer on RocksDB
commit 257b53d3b6df967f8a015a06c9e178d4219d0f8c
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T23:15:08Z
dummy
commit 25fd73107c577ac2e4b32300d4fe132ad7ff7312
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T23:21:29Z
merge from trunk
commit e57abf3c117d4ea7398c6157983d75443194ce9f
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T23:29:59Z
merge from demo example changes again
commit a0caee14d4a7b77045d9813a1072143db0fc8fa1
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-22T23:50:19Z
resolve conflicts
commit 5501e6fef16fdef55cb2136d8c3f8035d394d79f
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-26T17:12:12Z
Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into
K3060
commit d73b594147d1b4e25a61497550f498b9f14822a0
Author: Guozhang Wang <wangg...@gmail.com>
Date: 2016-01-26T19:04:11Z
move logging to rocksDBStore
----
> Refactor MeteredXXStore
> -----------------------
>
> Key: KAFKA-3060
> URL: https://issues.apache.org/jira/browse/KAFKA-3060
> Project: Kafka
> Issue Type: Sub-task
> Components: kafka streams
> Affects Versions: 0.9.0.1
> Reporter: Yasuhiro Matsuda
> Priority: Minor
>
> ** copied from a github comment by Guozhang Wang **
> The original motivation of having the MeteredXXStore is to wrap all metrics /
> logging semantics into one place so they do not need to be re-implemented
> again, but this seems to be an obstacle with the current pattern now, for
> example MeteredWindowStore.putAndReturnInternalKey is only used for logging,
> and MeteredWindowStore.putInternal / MeteredWindowStore.getInternal are never
> used since only its inner will trigger this function. So how about
> refactoring this piece as follows:
> 1. WindowStore only expose two APIs: put(K, V) and get(K, long).
> 2. Add a RollingRocksDBStores that does not extend any interface, but only
> implements putInternal, getInternal and putAndReturnInternalKey that uses
> underlying RocksDBStore<byte[], byte[]> as Segments.
> 3. RocksDBWindowStore implements WindowStore with an RollingRocksDBStores
> inner.
> 4. Let MeteredXXStore only maintain the metrics recording logic, and let
> different stores implement their own logging logic, since this is now
> different across different types and are better handled separately. Also some
> types of stores may not even have a loggingEnabled flag, if it will always
> log, or will never log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
