Thanks, Eno. On Tue, Jun 21, 2016 at 2:22 AM, Eno Thereska <eno.there...@gmail.com> wrote:
> Hi Roger, > > I realised I never got back to you on this one, sorry. Some answers inline: > > > On 3 Jun 2016, at 22:48, Roger Hoover <roger.hoo...@gmail.com> wrote: > > > > Hi Eno, > > > > Does this mean that Kafka Streams will disable the RocksDB write buffer? > > For the purposes of this KIP we might not want to change the current > RocksDB state of things. However, Guozhang has written about future plans > in the memory management page < > https://cwiki.apache.org/confluence/display/KAFKA/Discussion:+Memory+Management+in+Kafka+Streams> > in the wiki. > > > > Is it currently safe to recover a Kafka Streams application after SIGKILL > > on the same machine? If not, will this make it safe to do so? > > > > If RocksDB is not flushed before offsets are commited in Kafka and is > > killed with SIGKILL, will the data in the write buffer be lost (since > Kafka > > Streams disables the transaction log)? That data will be present in the > > Kafka changelog but will it get applied to the recovered RocksDB database > > on restart? > > These are good questions on failure modes. This KIP will not change the > failure behaviour from what it currently is. I believe there will be > subsequent KIPs where the issues around writing atomically to multiple > places will be considered, and failure semantics will be strengthened. Stay > tuned. > > Thanks > Eno > > > > > > > Thanks, > > > > Roger > > > > On Fri, Jun 3, 2016 at 2:39 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > > > >> Hi Gwen, > >> > >> Yes. As an example, if cache.max.bytes.buffering set to X, and if users > >> have A aggregation operators and T KTable.to() operators, then X*(A + T) > >> total bytes will be allocated for caching. > >> > >> Eno > >> > >>> On 3 Jun 2016, at 21:37, Gwen Shapira <g...@confluent.io> wrote: > >>> > >>> Just to clarify: "cache.max.bytes.buffering" is per processor? > >>> > >>> > >>> On Thu, Jun 2, 2016 at 11:30 AM, Eno Thereska <eno.there...@gmail.com> > >> wrote: > >>>> Hi there, > >>>> > >>>> I have created KIP-63: Unify store and downstream caching in streams > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams > >> < > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-63:+Unify+store+and+downstream+caching+in+streams > >>> > >>>> > >>>> > >>>> Feedback is appreciated. > >>>> > >>>> Thank you > >>>> Eno > >> > >> > >
