the other instance?
>
> Thanks,
> Damian
>
> On Fri, 15 Sep 2017 at 07:19 dev loper wrote:
>
> > Dear Kafka Users,
> >
> > I am fairly new to Kafka Streams . I have deployed two instances of Kafka
> > 0.11 brokers on AWS M3.Xlarge insatnces. I have created a topi
Dear Kafka Users,
I have updated the question in stackoverlflow . Please let me know about
any possible solution.
https://stackoverflow.com/questions/46233138/kafka-streams-application-unable-to-horizontally-scale-and-the-application-on-ot
On Fri, Sep 15, 2017 at 2:57 PM, dev loper wrote
t;
> I also noticed that one of your instances looks like it is configured with
> 12 threads, while the others have 4 - is that correct?
>
> On Fri, 15 Sep 2017 at 10:27 dev loper wrote:
>
> > Hi Damian,
> >
> > I do have the logs for the other application. But its kind
> Hi,
>
> Could you set MAX_POLL_RECORDS to something lower like 500 or 1000 and try
> again?
>
> Thanks,
> Bill
>
> On Fri, Sep 15, 2017 at 3:40 PM, dev loper wrote:
>
> > Hi Damian,
> >
> > I have repeated my tests with slight configuration
rdinator.java:725)
>
> Reducing MAX_POLL_RECORDS_CONFIG may lessen the symptom but that would
> not meet your throughput goal.
>
> On Fri, Sep 15, 2017 at 8:09 PM, dev loper wrote:
>
>> Hi All ,
>>
>> @Bill,
>>
>> I will reduce the MAX_POLL_RECORDS to 50
disconnected
java.io.EOFException
On Sat, Sep 16, 2017 at 1:04 PM, dev loper wrote:
> Thank you Ted for your inputs. I have also observed the similarity in
> logs. But I am not sure how to resolve the issue.
>
> @Bill, I repeated my tests by reducing the max MAX_POLL_RECORDS to 500. I
>
Hi Kafka Streams Users,
I am trying to improve the performance of Kafka Streams State Store
Persistent Store. In our application we are using Kafka Streams Processor
API and using Persistent State Store.. My application when starts up it
performing well but over a period of time the performance d
tore ?
>
> Are you using 0.11.0.0 ?
>
> Thanks
>
> On Sat, Sep 16, 2017 at 6:14 AM, dev loper wrote:
>
> > Hi Kafka Streams Users,
> >
> > I am trying to improve the performance of Kafka Streams State Store
> > Persistent Store. In our application we
s in your email had the same name)
> >
> > Is it possible that you were hitting KAFKA-5397 ?
> >
> > On Fri, Sep 15, 2017 at 12:40 PM, dev loper wrote:
> >
> >> Hi Damian,
> >>
> >> I have repeated my tests with slight configuration change. The
` call. It might be worth retrying at that point.
>
> Thanks,
> Bill
>
> On Sat, Sep 16, 2017 at 9:14 AM, dev loper wrote:
>
> > Hi Kafka Streams Users,
> >
> > I am trying to improve the performance of Kafka Streams State Store
> > Persistent Store. In our
I do remember it was at
> least 3x slower when compared to hashmap storage. Every lookup, every write
> contributed to the degradation. Changelogging had negligible overhead btw.
>
> Regards
> Sab
>
> On 17 Sep 2017 8:52 am, "dev loper" wrote:
>
> > Hi Bill.
>
d see if some oom killer or
> something similar is playing truant.
>
> I assume you have already validated with max poll records set to 500.
>
> Regards
> Sab
>
> On 15 Sep 2017 11:49 am, "dev loper" wrote:
>
> > Dear Kafka Users,
> >
> > I am fairly new
ntinue working on
> the State Store thread.
>
> Thanks!
> Bill
>
>
>
> On Mon, Sep 18, 2017 at 7:27 AM, dev loper wrote:
>
> > Hi Ted, Damian, Bill & Sabarish,
> >
> > I would like to thank you guys for all the help offered to solve this
&g
onfigs like so:
>
> props.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
> RocksDbLogsConfig.class);
>
>
>
> Thanks
> Bill
>
> On Sat, Sep 16, 2017 at 11:22 PM, dev loper wrote:
>
> > Hi Bill.
> >
> > Thank you pointing out, But in actual
happening for other partitions even though the
producer is sending message to the partitions where it was stuck ? Any help
here is very much appreciated.
Thanks
Dev Loper
}
>
>public static void main(String[] args) throws InterruptedException {
>MyKafkaConumer.INSTANCE.start();
>MyKafkaConumer.INSTANCE.join();
> }
> }
>
> I only have a single Consumer with a single thread running . What could be
> the reason for the Kafka Consumer to stop processing from certain
> partitions while the processing is happening for other partitions even
> though the producer is sending message to the partitions where it was
> stuck ? Any help here is very much appreciated.
>
> Thanks
> Dev Loper
>
>
>
>
>
client .
Thanks
Dev Loper
On Fri, Jul 6, 2018 at 11:01 PM, dev loper wrote:
> Hi Kafka Streams Users,
>
> I have posted the same question on stackoverflow and if anybody could
> point some directions it would be of great help.
>
> https://stackoverflow.com/questions/51214506
understand why this issue persists and what could be the reason. Any
pointer on how to debug this issue would be highly helpful .
On Mon, Jul 9, 2018 at 11:43 AM, Steve Tian wrote:
> Are you sure your consumer was still the owner of your partitions?
>
> On Mon, Jul 9, 2018, 12:54 PM
Hi Steve,
When the issue occurs I could see the lag shown for the corresponding
partitions in the output of kafka-consumer-groups command . I will try
changing the log level to debug for kafka consumer and I will post if
anything seems to be outside the normal range
Thank you
Dev Loper
On Mon
The processing does not take much time since the application reads from the
kafka and immediately puts into a in memory queue and another thread
persist these records into DB.
On Mon, Jul 9, 2018 at 12:44 PM, dev loper wrote:
> Hi Steve,
>
> When the issue occurs I could see the lag
20 matches
Mail list logo
