If I may use the same thread to discuss the exact same issue
Assuming one can store the offset in an external location (redis/db etc), along
with the rest of the state that a program requires, wouldn't it be possible to
manage things such that, you use the High Level API with auto commit t
Hi Folks,
At the recent Kafka Meetup in Mountain View there was interest expressed
about the encryption through Kafka proof of concept that Symantec did a
few months ago, so I have created a blog post with some details about it.
You can find that here:
http://goo.gl/sjYGWN
Let me know if you ha
Looks like you only have 4 messages in your topic and no more messages got
sent
2015-02-19 20:09:34,661] DEBUG initial fetch offset of consolemm:0: fetched
offset = 4: consumed offset = 4 is 4 (kafka.consumer.PartitionTopicInfo
You can try sending more messages to topic or give the MM a different
Yeah that is a good point - will do the update as part of the doc
changes in KAFKA-1729
On Thu, Feb 19, 2015 at 09:26:30PM -0500, Evan Huus wrote:
> On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote:
>
> > If you are using v0 of OffsetCommit/FetchRequest then you can issue
> > that to any broker
On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote:
> If you are using v0 of OffsetCommit/FetchRequest then you can issue
> that to any broker. For version > 0 you will need to issue it to the
> coordinator. You can discover the coordinator by sending a
> ConsumerMetadataRequest to any broker.
>
If you are using v0 of OffsetCommit/FetchRequest then you can issue
that to any broker. For version > 0 you will need to issue it to the
coordinator. You can discover the coordinator by sending a
ConsumerMetadataRequest to any broker.
On Thu, Feb 19, 2015 at 07:55:16PM +, Suren wrote:
> Joel/A
The log end offset (of a partition) changes when messages are appended
to the partition. (It is not correlated with the consumer's offset).
On Thu, Feb 19, 2015 at 08:58:10PM +, Will Funnell wrote:
> So at what point does the log end offset change? When you commit?
>
> On 19 February 2015 at
Yes it is supported in 0.8.2-beta. It is documented on the site - you
will need to set offsets.storage to kafka.
On Thu, Feb 19, 2015 at 03:57:31PM -0500, Matthew Butt wrote:
> I'm having a hard time figuring out if the new Kafka-based offset
> management in the high-level Scala Consumer is implem
Jun,
I am already using the latest release 0.8.2.1.
-Zakee
On Thu, Feb 19, 2015 at 2:46 PM, Jun Rao wrote:
> Could you try the 0.8.2.1 release being voted on now? It fixes a CPU issue
> and should reduce the CPU load in network thread.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 19, 2015 at 11:54 AM,
[2015-02-05 14:21:09,708] ERROR [ReplicaFetcherThread-2-1], Error in fetch
Name: FetchRequest; Version: 0; CorrelationId: 147301; ClientId:
ReplicaFetcherThread-2-1; ReplicaId: 3; MaxWait: 500 ms; MinBytes: 1 bytes;
RequestInfo: [site.db.people,6] ->
PartitionFetchInfo(0,1048576),[site.db.main,4] -
Is there any error in the producer log? Is there any pattern in the
messages being lost?
Thanks,
Jun
On Thu, Feb 19, 2015 at 4:20 PM, Karts wrote:
> yes i did.
>
> On Thu, Feb 19, 2015 at 2:42 PM, Jun Rao wrote:
>
> > Did you consume the messages from the beginning of the log?
> >
> > Thanks,
actually i take that back. it reads from where the last offset left off.
On Thu, Feb 19, 2015 at 4:20 PM, Karts wrote:
> yes i did.
>
> On Thu, Feb 19, 2015 at 2:42 PM, Jun Rao wrote:
>
>> Did you consume the messages from the beginning of the log?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Feb 19,
yes i did.
On Thu, Feb 19, 2015 at 2:42 PM, Jun Rao wrote:
> Did you consume the messages from the beginning of the log?
>
> Thanks,
>
> Jun
>
> On Thu, Feb 19, 2015 at 12:18 PM, Karts wrote:
>
> > but they have always been up. I mean when i was testing, all the
> zookeepers
> > were up. and al
Hey Jun,
That's what I've got right now, semaphore before send() and release in the
callback. Am I correct in understanding that there's no way to do any
batching with KafkaProducer itself (other than have a "bulk" message which
would just be a single message with multiple messages for a particula
You can register a callback for each message sent. The callback will be
called when the message is sent successfully or failed.
Thanks,
Jun
On Tue, Feb 17, 2015 at 4:11 PM, JAmes Atwill
wrote:
> Hi!
>
> I'm using the new KafkaProducer in 0.8.2.0.
>
> I have thousands of "Nodes" which receive m
Did you consume the messages from the beginning of the log?
Thanks,
Jun
On Thu, Feb 19, 2015 at 12:18 PM, Karts wrote:
> but they have always been up. I mean when i was testing, all the zookeepers
> were up. and all the kafka nodes were up. its just that I changed the
> number of zookeeper nod
Could you try the 0.8.2.1 release being voted on now? It fixes a CPU issue
and should reduce the CPU load in network thread.
Thanks,
Jun
On Thu, Feb 19, 2015 at 11:54 AM, Zakee wrote:
> Kafka documentation recommends <0.3 for above metric. I assume processor is
> busier if this goes below 0.3
Jun,
You are right. I tried 0.8.2.0 producer with my test. confirmed that it
fixed the cpu issue.
Thanks,
Steven
On Thu, Feb 19, 2015 at 12:02 PM, Steven Wu wrote:
> will try 0.8.2.1 on producer and report back result.
>
> On Thu, Feb 19, 2015 at 11:52 AM, Jun Rao wrote:
>
>> This is probabl
I'm having a hard time figuring out if the new Kafka-based offset
management in the high-level Scala Consumer is implemented in the current
version of 0.8.2-beta. If I implement a high-level consumer, will it use
the new system, or will it still be storing in zookeeper? Do I need to wait
for the Ja
So at what point does the log end offset change? When you commit?
On 19 February 2015 at 18:47, Joel Koshy wrote:
> > If I consumed up to the log end offset and log compaction happens in
> > between, I would have missed some messages.
>
> Compaction actually only runs on the rolled over segments
but they have always been up. I mean when i was testing, all the zookeepers
were up. and all the kafka nodes were up. its just that I changed the
number of zookeeper nodes in my first test iteration. second and third were
still the same. not sure why the topics were losing some messages.
On Thu, F
Tao,
I updated the mirrorconsumer.properties config file as you suggested, and
upped the MM's log level to DEBUG. I have the output of the DEBUG logger
here in this pastebin, if you could take a minute to look for anything in
its contents that would indicate a problem that would be extremely help
Any error in the controller and state-change log?
Thanks,
Jun
On Thu, Feb 12, 2015 at 7:28 AM, Omid Aladini wrote:
> Hi,
>
> I'm experimenting with the following scenario:
>
> - 3 brokers are running (0,1 and 2) -- Kafka version 0.8.2.0
> - Continuously: restart broker number 0 by triggering c
will try 0.8.2.1 on producer and report back result.
On Thu, Feb 19, 2015 at 11:52 AM, Jun Rao wrote:
> This is probably due to KAFKA-1642, which is fixed in 0.8.2.0. Could you
> try that version or 0.8.2.1 which is being voted now.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 19, 2015 at 10:42 AM, Steve
Kafka documentation recommends <0.3 for above metric. I assume processor is
busier if this goes below 0.3 and obviously it being < 0.3 for long does
not seem to be a good sign.
What should be our criteria to raise an alert, I though it should be when
its value goes below 0.3. However, the value s
Joel/All,
The SimpleConsumer constructor requires a specific host and port.
Can this be any broker?
If it needs to be a specific broker, for 0.8.2, should this be the offset
coordinator? For 0.8.1, does it matter?
-Suren
On Thursday, February 19, 2015 10:43 AM, Joel Koshy
wrote:
I
This is probably due to KAFKA-1642, which is fixed in 0.8.2.0. Could you
try that version or 0.8.2.1 which is being voted now.
Thanks,
Jun
On Thu, Feb 19, 2015 at 10:42 AM, Steven Wu wrote:
> forgot to mention in case it matters
> producer: 0.8.2-beta
> broker: 0.8.1.1
>
> On Thu, Feb 19, 2015
Zookeeper requires a majority of the nodes to be up for the service to be
available. Kafka relies on Zookeeper to be always available.
Thanks,
Jun
On Thu, Feb 19, 2015 at 11:15 AM, Karts wrote:
> I have noticed some strange patterns when testing with the 0.8.1 build and
> the 0.8.2 builds, and
+1 binding.
Checked the md5, and quick start.
Some minor comments:
1. The quickstart section would better include the building step after
download and before starting server.
2. There seems to be a bug in Gradle 1.1x with Java 8 causing the "gradle"
initialization to fail:
-
FAILURE:
I have noticed some strange patterns when testing with the 0.8.1 build and
the 0.8.2 builds, and are listed below.
1. So I setup a brand new cluster [3 kafka nodes with 3 zookeepers],
created 2 topics via the API calls, everything went fine and was
successfully able to view my messages in my consum
Well are there any measurement techniques for Memory config in brokers. We
do have a large load, with a max throughput 200MB/s. What do you suggest as
the recommended memory config for 5 brokers to handle such loads?
On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps wrote:
> 40G is really huge, general
> If I consumed up to the log end offset and log compaction happens in
> between, I would have missed some messages.
Compaction actually only runs on the rolled over segments (not the
active - i.e., latest segment). The log-end-offset will be in the
latest segment which does not participate in com
forgot to mention in case it matters
producer: 0.8.2-beta
broker: 0.8.1.1
On Thu, Feb 19, 2015 at 10:34 AM, Steven Wu wrote:
> I think this is an issue caused by KAFKA-1788.
>
> I was trying to test producer resiliency to broker outage. In this
> experiment, I shutdown all brokers and see how pr
I think this is an issue caused by KAFKA-1788.
I was trying to test producer resiliency to broker outage. In this
experiment, I shutdown all brokers and see how producer behavior.
Here are the observations
1) kafka producer can recover from kafka outage. i.e. send resumed after
brokers came back
> The log end offset is just the end of the committed messages in the log
> (the last thing the consumer has access to). It isn't the same as the
> cleaner point but is always later than it so it would work just as well.
Isn't this just roughly the same value as using c.getOffsetsBefore() with a
p
Hello,
We're having the following issue with Kafka and/or Zookeeper:
If a broker (id=1) is running, and you start another broker with id=1, the new
broker will exit saying "A broker is already registered on the path
/brokers/ids/1". However, I noticed when I query zookeeper /brokers/ids/1
disap
The log end offset is just the end of the committed messages in the log
(the last thing the consumer has access to). It isn't the same as the
cleaner point but is always later than it so it would work just as well.
-Jay
On Thu, Feb 19, 2015 at 8:54 AM, Will Funnell wrote:
> > I'm not sure if I
> I'm not sure if I misunderstood Jay's suggestion, but I think it is
> along the lines of: we expose the log-end-offset (actually the high
> watermark) of the partition in the fetch response. However, this is
> not exposed to the consumer (either in the new ConsumerRecord class
> or the existing M
I see - yes, you can use the SimpleConsumer for that. However, your
high-level consumers need to be shutdown while you do that (otherwise
they may auto-commit while you are resetting offsets).
Thanks,
Joel
On Thu, Feb 19, 2015 at 03:29:19PM +, Suren wrote:
> We are using the High Level Consu
We are using the High Level Consumer API to interact with Kafka for our normal
use cases.
However, on consumer restart in the case of consumer failures, we want to be
able to manually
reset offsets in certain situations.
And ideally we'd like to use the same api in 0.8.1 and 0.8.2. :-)
It looked
Not sure what you mean by using the SimpleConsumer on failure
recovery. Can you elaborate on this?
On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote:
> Haven't used either one now. Sounds like 0.8.2.1 will help.
> We are using the High Level Consumer generally but are thinking to use the
> Si
Haven't used either one now. Sounds like 0.8.2.1 will help.
We are using the High Level Consumer generally but are thinking to use the
SimpleConsumer on failure recovery to set the offsets.
Is that the recommended approach for this use case?
Thanks.
-Suren
On Thursday, February 19, 2015 9:
Are you using it from Java or Scala? i.e., are you using the
javaapi.SimpleConsumer or kafka.consumer.SimpleConsumer
In 0.8.2 javaapi we explicitly set version 0 of the
OffsetCommitRequest/OffsetFetchRequest which means it will
commit/fetch to/from ZooKeeper only. If you use the scala API you can
Joel,
Looking at SimpleConsumer in the 0.8.2 code, it is using OffsetCommitRequest
and sending that over to a broker.
Is the broker storing that in ZK?
-Suren
On Tuesday, February 17, 2015 12:22 PM, Joel Koshy
wrote:
Hi Chris,
In 0.8.2, the simple consumer Java API supports commit
thanks mani for quick response, sorry some how i missed this javadoc :)
t
SunilKalva
On Thu, Feb 19, 2015 at 6:14 PM, Manikumar Reddy
wrote:
> Hi,
>
> In new producer, we can specify the partition number as part of
> ProducerRecord.
>
> From javadocs :
> *"If a valid partition number is specifi
Hi,
In new producer, we can specify the partition number as part of
ProducerRecord.
>From javadocs :
*"If a valid partition number is specified that partition will be used when
sending the record. If no partition is specified but a key is present a
partition will be chosen using a hash of the key
> Hi
> I could not find a way to customize "Partitioner" class in new
> KafaProducer class, is it intentional ?
>
> tx
>
SunilKalva
>
Hi
I could not find a way to customize "Partitioner" class in new KafaProducer
class, is it intentional ?
tx
SunilKalva
48 matches
Mail list logo
