Hi,
I am building kafka client using C++ Client API. The offsetRequest API is based
on
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetRequest
According to the documentation, the “Time” field in offsetRequest is used
Hi all
i use 0.8.1 release to do test, and get following error logs
[hadoop@nelo76 libs]$ [2014-03-14 12:11:44,310] INFO Partition
[nelo2-normal-logs,0] on broker 0: Shrinking ISR for partition
[nelo2-normal-logs,0] from 0,1 to 0 (kafka.cluster.Partition)
[2014-03-14 12:11:44,313] ERRO
If you set auto.offset.reset to smallest, the consumer should pick up the
first message on new topics.
Thanks,
Jun
On Thu, Mar 13, 2014 at 2:42 AM, David Morales de Frías <
dmora...@paradigmatecnologico.com> wrote:
> Hi there,
>
> A few days ago i have written about an issue (0.8.0) when auto-
There are some known bugs with the partition reassignment tool in 0.8. We
fixed a lot of those, tested it on large set of partitions and found it to
be stable in 0.8.1. Could you give 0.8.1 a spin instead?
On Thu, Mar 13, 2014 at 4:05 PM, Ryan Berdeen wrote:
> After reassigning a topic to 5 new
After reassigning a topic to 5 new brokers, the topic metadata returned by
the brokers for the topic is causing an exception when a producer tries to
read it:
java.util.NoSuchElementException: key not found: 7
...
at kafka.api.PartitionMetadata$.readFrom(TopicMetadata.scala:104)
The reassignment
Thanks for the response,
indeed the encryption/concealment should ideally be done on the producer
side but it's just not feasible in some of our applications so a middle
layer would be needed. So far our thoughts have been around using Flume
interceptors but that means introducing another "moving
I see. In that case, it might be easier to write a MockSimpleConsumer and
have it return different FetchResponse objects with the error codes you
want to test. We don't have a MockSimpleConsumer as well. If you happen to
write one, you could use it with EasyMock to get the fetch() API to return
the
Neha. That is correct SimpleConsumer does not do much, but my code that is
calling it need to react properly at error response and I wanted to test
that.
On Thu, Mar 13, 2014 at 1:21 PM, Neha Narkhede wrote:
> Currently, we don't have a mock class for the server. The SimpleConsumer
> actually do
Or introduce an app layer between the producers and kafka that does the
processing without changes/load to the producers.
On Thu, Mar 13, 2014 at 1:18 PM, Neha Narkhede wrote:
> In general, the preference has been to avoid having user code run on the
> brokers since that just opens a can of worm
I think it is worth fixing
https://issues.apache.org/jira/browse/KAFKA-1006since the behavior for
new topic discovery is broken in the new consumer.
Even if the consumer sets auto.offset.reset=largest, it is very unintuitive
to apply that behavior to the discovery of new topics. I think that new
to
Currently, we don't have a mock class for the server. The SimpleConsumer
actually doesn't do much with the fetch request and response. Take a look
at the fetch() API on SimpleConsumer. It merely returns the response back
to the user.
Thanks,
Neha
On Wed, Mar 12, 2014 at 11:58 PM, Vadim Keylis wr
In general, the preference has been to avoid having user code run on the
brokers since that just opens a can of worms where the broker logic get's
complicated trying to deal with errors that the user code can throw. The
suggestion is to push any user specific processing to the client side. In
this
Hi,
I have a use case for which it would be useful with pluggable processing
functions in the broker.
We have some data containing sensitive information which is legally ok to
transmit over the internal network to the Kafka brokers and keep in
volatile memory but not to flush to disk unconcealed/
Hi,
I have a use case that might be relevant to the new consumer API but that
would require most of the work on the broker. I would be surprised if it
had not been discussed before but I was not able to find any directly
related thread. Has there been any discussion about providing broker side
con
Hi David,
This issue should only happen if the following conditions are satisfied
1) The consumer does not have any offset stored in ZK for this topic when
it first fetches messages for it.
2) The consumer's default offset reset value is "latest".
3) After the topic is created, it will take a bit
Heh, it's not a bad link--you just need to run the docs on a local web
server like me. Doesn't everyone do that? :-)
Fixed it, thanks!
-Jay
On Thu, Mar 13, 2014 at 9:44 AM, Roger Hoover wrote:
> Thanks, Jay. Great write up.
>
> I noticed a bad link to the docs for basic operations (
> http://
Thanks, Jay. Great write up.
I noticed a bad link to the docs for basic operations (
http://localhost/documentation.html#basic_ops). It's in the paragraph that
starts with "We also improved a lot of operational activities...".
Roger
On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps wrote:
> Hi guy
One system I am working on we have something similar however we are
creating topics like how the TopicCommand does but different. We keep
those operations persisted in a meta store so consumers that have to-do
something know that everything is all ready for them (as often it is not
just Kafka) for
Hi,
If someone is interested, I found this projet
https://github.com/wikimedia/operations-debs-kafka from wikimedia.
If you want to build it, you can use git-buildpackage or you can find the
prebuilt .deb package there :
http://apt.wikimedia.org/wikimedia/pool/universe/k/kafka/
François Langel
Hello Guozhang,
We are using Kafka to connect several actors in our project (osgi, storm,
vert.x web app...), all of them being consumers and producers of different
topics. These services can go up and down in different moments.
Anyway, the issue is the same when you first start the consumer and
Hello David,
This is a known issue with consumer starting before the topic is created:
https://issues.apache.org/jira/browse/KAFKA-1006
And we are still figuring the right way to fix it. Could you describe your
use cases where you need to start the consumer beforehand?
Guozhang
On Thu, Mar 13
Thanks for the writeup. In 0.8.1, we also significantly improved the
partition reassignment tool.
Jun
On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps wrote:
> Hi guys,
>
> I wrote up a quick blog post on the new stuff in 0.8.1:
>
> http://blog.empathybox.com/post/79427855885/whats-new-in-kafka-0-8-
Hi there,
A few days ago i have written about an issue (0.8.0) when auto-create topic
is enabled and consumer starts before producer (consumer never sees any
message and they seem to be lost even by the brokers)
Now i have done the same test with 0.8.1 and the issue is partially fixed
but now the
That kafkaServer-gc.log contains GC logs produced by the JVM. The JVM GC
logging options are set in kafka-run-class.sh script from Kafka
distribution. GC logging is enabled by default. You should be able to
override the default behavior with KAFKA_GC_LOG_OPTS variable (never tried
it myself though)
Hi
I am using kafka_2.8.0-0.8.0.tar.gz bundle.
I have set Log level as "OFF" in log4j.properties file and according to
this log level,the logging is off(there should be no log generation in any
.log file).
But Kafka is generating logs in kafkaServer-gc.log file.Please tell if
logging of this par
25 matches
Mail list logo
