i found the problem by myself. since i don't know nothing about JMX and the
Beans, i was querying the bean wrong. i still don't know how to query the
specific topic metrics in jconsole. but it works for me to query (with
jolokia) instead of:
kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics
Hi team,
I found that the producer metric compression-rate-avg always returns 0 even
with compression.type set to snappy. I drilled down the code and discovered
that the position of bytebuffer in
org.apache.kafka.common.record.Compressor is reset to 0 by
RecordAccumulator.drain() before calling me
Hi Jason,
Thanks for your response. See replies inline:
On Tuesday, December 15, 2015, Jason Gustafson wrote:
> Hey Jens,
>
> I'm not sure I understand why increasing the session timeout is not an
> option. Is the issue that there's too much uncertainly about processing
> time to set an upper b
Yes, please.
Ismael
On 15 Dec 2015 11:52, "tao xiao" wrote:
> Hi team,
>
> I found that the producer metric compression-rate-avg always returns 0 even
> with compression.type set to snappy. I drilled down the code and discovered
> that the position of bytebuffer in
> org.apache.kafka.common.reco
Hey Jens,
The purpose of pause() is to stop fetches for a set of partitions. This
lets you continue calling poll() to send heartbeats. Also note that poll()
generally only blocks for rebalances. In code, something like this is what
I was thinking:
while (running) {
ConsumerRecords records = con
Dave,
This is a new app being developed to use the Bluemix Messaging Hub 0.9 beta
service. No migration involved from 0.8.
With batching working for us (having fixed our own coding bug), we’re quite
pleased with the numbers we’re seeing.
Our mindset is that in using a cloud service solution,
When using the 0.8.2 high level consumer, what is the impact of creating many
one-time use groupIds and checkpointing offsets using those?
I have a use case where upon every boot, I want to consume an entire topic from
the very beginning, all partitions. We are using the high level consumer for
We are noticing into these errors against certain partitions (4 out of 12
partitions) on the consumer and keeps filling up the logs:
ERROR consumer.ConsumerFetcherThread: [ConsumerFetcherThread-],
Current offset 104088851 for partition [some-topic,5] out of range;
reset offset to 104088851
I
We want to be able to monitor the ability to send messages to Kafka topics. We
want to be aware of the inability to do so before the time we attempt to send a
message. What we're looking for is something like a heartbeat. The reason we
need this is that in our deployment environment, Kafka an
I also notice that the errors are for the partitions that seem to be under
replicated. We got 3 brokers and one of the brokers never seems to be
chosen as leader for any of the 12 partitions of the topic.
On Tue, Dec 15, 2015 at 11:24 AM, Buntu Dev wrote:
> We are noticing into these errors agai
We had to revert to 0.8.3 because three of our topics seem to have gotten
corrupted during the upgrade. As soon as we did the upgrade producers to
the three topics I mentioned stopped being able to do writes. The clients
complained (occasionally) about leader not found exceptions. We restarted
our
We are trying to use the Kafka 0.9 consumer API to poll specific
partitions. We consume partitions based on our own logic instead of
delegating that to Kafka. One of our use cases is handling a change in the
partitions that we consume. This means that sometimes we need to consume
additional partiti
Hey Rajiv,
I agree the Set/List inconsistency is a little unfortunate (another
annoying one is pause() which uses a vararg). I think we should probably
add the following variants:
assign(Collection)
subscribe(Collection)
pause(Collection)
I can open a JIRA to fix this. As for returning the unmod
Fabian,
Have a look at jmxc - it's a simple command-line tool we open-sourced ages
ago. Very handy for dumping the whole JMX content, which helps when you
don't know what exactly you are looking for :)
https://github.com/sematext/jmxc
Otis
--
Monitoring - Log Management - Alerting - Anomaly Det
Hi Jason,
The copying is not a problem in terms of performance. It's just annoying to
write the extra code. My point with the copy is that since the client is
already making a copy when it returns the set to me, why would it matter if
I modify the copy. Creating an unmodifiable set on top of a cop
Hey Rajiv,
My point was that you could maintain the assignment set yourself in a
field, which would eliminate the need to copy the set returned by
assignment(). Then it's just one copy to convert it to a list, and we can
fix this by adding the assign() variant I suggested above.
By the way, here'
Right I could do that. Thanks for creating the JIRA!
On Tue, Dec 15, 2015 at 3:01 PM, Jason Gustafson wrote:
> Hey Rajiv,
>
> My point was that you could maintain the assignment set yourself in a
> field, which would eliminate the need to copy the set returned by
> assignment(). Then it's just
Hey Rajiv,
Are you using snappy compression?
On Tue, Dec 15, 2015 at 12:52 PM, Rajiv Kurian wrote:
> We had to revert to 0.8.3 because three of our topics seem to have gotten
> corrupted during the upgrade. As soon as we did the upgrade producers to
> the three topics I mentioned stopped being
created https://issues.apache.org/jira/browse/KAFKA-2993. I will submit a
patch for this
On Wed, 16 Dec 2015 at 00:53 Ismael Juma wrote:
> Yes, please.
>
> Ismael
> On 15 Dec 2015 11:52, "tao xiao" wrote:
>
> > Hi team,
> >
> > I found that the producer metric compression-rate-avg always return
I was talking with Jay this afternoon about this use case. The tricky thing
about adding a ping() or heartbeat() API is that you have to deal with the
potential for rebalancing. This means either allowing it to block while a
rebalance completes or having it raise an exception indicating that a
reba
20 matches
Mail list logo
