Re: Kafka Streams Disk Usage on upgrade to 2.1.0

2019-03-05 Thread Adrian McCague
Thanks Bill, I have written up a ticket here: https://issues.apache.org/jira/browse/KAFKA-8042 Adrian On 05/03/2019, 15:44, "Bill Bejeck" wrote: Hi Adrian, No, it's not an expected outcome. Could you file a Jira ticket and include the information requested by Guozha

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

2019-03-05 Thread Bill Bejeck
Hi Adrian, No, it's not an expected outcome. Could you file a Jira ticket and include the information requested by Guozhang (code and configs) and we can try to reproduce the error? Thanks, Bill On Tue, Mar 5, 2019 at 10:14 AM Adrian McCague wrote: > Drilling down further: > > bash-4.2# pwd >

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

2019-03-05 Thread Adrian McCague
Drilling down further: bash-4.2# pwd /data/fooapp/0_7 bash-4.2# for dir in $(find . -maxdepth 1 -type d); do echo "${dir}: $(find ${dir} -type f -name 'MANIFEST-*' -printf x | wc -c)"; done .: 8058 ./KSTREAM-JOINOTHER-25-store: 851 ./KSTREAM-JOINOTHER-40-store: 819 ./KSTREAM-JOINT

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

2019-03-01 Thread Adrian McCague
Hi Guozhang, thanks for your response. I have done some further investigations. The difference I see between the two versions is the following, in 1.1 this is the stat of the rocksdb MANIFEST files of one of the partitions: root@fooapp-6c4649dd68-wzrxk:/data# stat fooapp/2_5/rocksdb/foo-store/

Re: Kafka Streams Disk Usage on upgrade to 2.1.0

2019-03-01 Thread Guozhang Wang
Hello Adrian, What you described did sounds wired to me. I'm not aware of any regressions on rocksDB disk usage from 1.1 to 2.1. Could you file a JIRA ticket with more details like state dir snapshots, your code snippet and configs etc so we can find a way to reproduce it? Guozhang On Fri, Mar