Have you checked the EBS burst balance on your disks that the streams
application is running on?
On 21 September 2017 at 04:28, dev loper wrote:
> Hi Bill,
>
> I will repeat my tests with Rocks DB enabled and I will revert to you with
> details. I might take 1-2 days to get back to you with deta
l
> but needs to be managed by you, as the user.
>
> We are working on fixing KAFKA-4601 soon, but this may involve a rather
> general fix, to refactor the DSL translation to go beyond
> operator-by-operator steps.
>
>
> Guozhang
>
> On Tue, Aug 29, 2017 at 3:59
e it
> would cause the Counts table to return `null` indicating it has not
> received a record yet.
>
>
> Guozhang
>
>
> On Thu, Aug 24, 2017 at 3:17 AM, Ian Duffy wrote:
>
> > Hi All,
> >
> > I'm building a streams applications where I wis
Hi All,
I'm building a streams applications where I wish to take action on the
input when a certain frequency of the input has been seen.
At the moment the application roughly goes:
frequency table = Input Stream -> groupByKey -> Count
input stream with counts = leftJoin frequency table and inp
Hi Jens,
We did something similar to this at Zalando.
https://github.com/zalando-incubator/remora
It effectively supplies the kafka consumer group supply command as a http
endpoint.
On 12 August 2017 at 16:42, Subhash Sriram wrote:
> Hi Jens,
>
> Have you looked at Burrow?
>
> https://github.
.
> >
> > Thanks,
> > Damian
> >
> > On Fri, 30 Jun 2017 at 16:38 Ian Duffy wrote:
> >
> >> Hi All,
> >>
> >> I was wondering if any of those who know stream internals should shed
> any
> >> light on the
Hi All,
I was wondering if any of those who know stream internals should shed any
light on the following exception:
org.apache.kafka.streams.errors.ProcessorStateException: Error while
closing the state manager at
org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(Abstrac
Hey Abhimanyu,
Not directly answering your questions but in the past we used burrow at my
current company and we had a horrible time with it. It would crash daily
and its lag metrics were very different to what was returned when you would
run the kafka-consumer-group describe command as you noted.
Hi All,
In the past my team was using burrow by linkedin for monitoring consumer
group lag. We discovered it crashed a lot and we found ourselves constantly
restarting it.
As a fix, a co-worker has created his own solution that I would like to
share with you https://github.com/zalando-incubator/r
Hi All,
In the past, we experienced lots of problems with running Kafka Stream
Applications on AWS.
We've seen issues with state locking, memory spiking to 100% and the
instance dying, very
slow startup on pulling down initial rocksdbs and so on
Today we realised that the instances were expe
Hi All,
Is there any community preferred tooling for doing point in time backups of
kafka? (ideally without downtime)
We've looked at https://github.com/pinterest/secor but refeeding ~500gb+ of
data doesn't seem too neat.
Thanks,
Ian.
+1
Started using kafka client 0.10.2.1 for our streams applications, seen a
much greater improvement on retries when failures occur.
We've been running without manual intervention for > 24 hours which is
something we haven't seen in awhile.
Found it odd that the RC tag wasn't within the version o
send it to me
> directly if you don't want to broadcast it everywhere.
>
> Thanks
> Eno
>
>
> > On 25 Apr 2017, at 17:36, Ian Duffy wrote:
> >
> > Thanks again for the quick response Eno.
> >
> > We just left the application running in the h
;
> Retries are sometimes expected and don't always indicate a problem. We
> should probably adjust the printing of the messages to not print this
> warning frequently. Are you seeing any crash or does the app proceed?
>
> Thanks
> Eno
>
> On 25 Apr 2017 4:02 p.m., "
ase to 10
> from
> >> default of 0
> >> props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG,
> >> Integer.toString(Integer.MAX_VALUE)); <- increase to infinity
> >> from default of 300 s
> >>
> >> Thanks
> >> Eno
> >>
ew Properties();
> ...
> props.put(ProducerConfig.RETRIES_CONFIG, 10); < increase to 10 from
> default of 0
> props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG,
> Integer.toString(Integer.MAX_VALUE)); <- increase to infinity
> from default of 300 s
>
> Thanks
> E
Hi All,
We're running multiple Kafka Stream applications using Kafka client
0.10.2.0 against a 6 node broker cluster running 0.10.1.1
Additionally, we're running Kafka Connect 0.10.2.0 with the ElasticSearch
connector by confluent [1]
On an ISR change occurring on the brokers, all of the streams
?
>
> Thanks
> Eno
>
>
> > On Mar 10, 2017, at 9:05 AM, Ian Duffy wrote:
> >
> > Hi All,
> >
> > I'm doing a groupBy and reduce on a kstream which results in a state
> store
> > being created.
> >
> > This state store is grow
Hi All,
I'm doing a groupBy and reduce on a kstream which results in a state store
being created.
This state store is growing to be massive, its filled up a 20gb drive. This
feels very unexpected. Is there some cleanup or flushing process for the
state stores that I'm missing or is such a large s
these things automatically). I'd suggest the scaling out approach first.
>
> I think the info you already have would be sufficient in creating the JIRA
> for now (i.e., feel free to copy paste your email)
>
> Thanks
> Eno
>
> > On 27 Feb 2017, at 14:53, Ian Duffy
u are using the latest release 0.10.2 (that just came out last week)?
> Just double-checking before re-opening the JIRA. As a work around, try
> setting num.stream.threads to 1 and run instead multiple
> instances/processes on the same machine.
>
> Thanks
> Eno
>
> > On 27
Hi All,
I'm using Kafka Client 10.2 with Kafka Streams.
I'm performing a groupByKey on a stream and seeing large files appear
within my state directory. Is this expected?
90M 1_0/rocksdb/content-count-store
82M 1_1/rocksdb/content-count-store
102M 1_10/rocksdb/content-count-store
86M 1_11/rocksd
Seeing the following failure when using multi-threaded streams
Feb 10 17:21:15 ip-172-31-137-57 docker/43e65fe123cd[826]:
org.apache.kafka.streams.errors.LockException: task [0_21] Failed to lock
the state directory: /tmp/kafka-streams/text_pipeline_id/0_21
Feb 10 17:21:15 ip-172-31-137-57 docker/
23 matches
Mail list logo
