Guozhang, I think the info is there. The next sentence says "The optimization is applicable to aggregations. It is not applicable to other operators like joins." Also the bit about the per-thread cache is one paragraph lower: "Specifically, for a streams instance with T threads and cache size C, each thread will have an even C/T bytes of cache, to use as it sees fit among its tasks. No sharing of caches across threads will happen"
Thanks Eno > On 8 Sep 2016, at 18:51, Guozhang Wang <wangg...@gmail.com> wrote: > > Just read the KIP wiki again: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams > > Just one minor comments but otherwise I'm +1: > > In "proposed changes" section: "The cache has two functions. First, it > continues to serve as a read buffer for data that is sent to the state > store, just like today. Second, it serves as a write deduplicator for the > state store (just like today) as well as for the downstream processor > node(s). " > > I feel this is still a bit confusing. The caching layer is only turned on > for state stores used in aggregate operators in the DSL, right? For example > in KStream-KStream joins, we will not turn on caching since there is no > updates on the state store on the same key. > > So could we just propose "removing the caching layer inside the persistent > state store engines (i.e. RocksDB), and instead add a per-thread global > cache which will only be activated for state stores used in Streams DSL's > aggregate operator as a write deduplicator for both the state store and to > the downstream operators"? > > Guozhang > > > On Thu, Sep 8, 2016 at 10:07 AM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> There have been a couple of changes to KIP-63 since the voting started, >> after more feedback, most notably the fact that this KIP applies to the DSL >> only, and not to the Processor API. >> >> At this point I'd like to restart the voting process. >> >> Thanks >> Eno >> >>> On 31 Aug 2016, at 17:16, Jim Jagielski <j...@jagunet.com> wrote: >>> >>> +1 >>>> On Aug 25, 2016, at 6:57 AM, Eno Thereska <eno.there...@gmail.com> >> wrote: >>>> >>>> Hi folks, >>>> >>>> We'd like to start the vote for KIP-63. At this point the Wiki addresses >>>> all previous questions and we believe the PoC is feature-complete. >>>> >>>> Thanks >>>> Eno >>> >> >> > > > -- > -- Guozhang
