Hi Roger, I realised I never got back to you on this one, sorry. Some answers inline:
> On 3 Jun 2016, at 22:48, Roger Hoover <roger.hoo...@gmail.com> wrote: > > Hi Eno, > > Does this mean that Kafka Streams will disable the RocksDB write buffer? For the purposes of this KIP we might not want to change the current RocksDB state of things. However, Guozhang has written about future plans in the memory management page <https://cwiki.apache.org/confluence/display/KAFKA/Discussion:+Memory+Management+in+Kafka+Streams> in the wiki. > Is it currently safe to recover a Kafka Streams application after SIGKILL > on the same machine? If not, will this make it safe to do so? > > If RocksDB is not flushed before offsets are commited in Kafka and is > killed with SIGKILL, will the data in the write buffer be lost (since Kafka > Streams disables the transaction log)? That data will be present in the > Kafka changelog but will it get applied to the recovered RocksDB database > on restart? These are good questions on failure modes. This KIP will not change the failure behaviour from what it currently is. I believe there will be subsequent KIPs where the issues around writing atomically to multiple places will be considered, and failure semantics will be strengthened. Stay tuned. Thanks Eno > > Thanks, > > Roger > > On Fri, Jun 3, 2016 at 2:39 PM, Eno Thereska <eno.there...@gmail.com> wrote: > >> Hi Gwen, >> >> Yes. As an example, if cache.max.bytes.buffering set to X, and if users >> have A aggregation operators and T KTable.to() operators, then X*(A + T) >> total bytes will be allocated for caching. >> >> Eno >> >>> On 3 Jun 2016, at 21:37, Gwen Shapira <g...@confluent.io> wrote: >>> >>> Just to clarify: "cache.max.bytes.buffering" is per processor? >>> >>> >>> On Thu, Jun 2, 2016 at 11:30 AM, Eno Thereska <eno.there...@gmail.com> >> wrote: >>>> Hi there, >>>> >>>> I have created KIP-63: Unify store and downstream caching in streams >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams >> < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63:+Unify+store+and+downstream+caching+in+streams >>> >>>> >>>> >>>> Feedback is appreciated. >>>> >>>> Thank you >>>> Eno >> >>
