Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the
parallelism isn't getting achieved as I have 3 partitions and 2 task for the
spout, but only one of it is emitting the data.
PFA the screen grab for number of task of spout and data emitted by only one of
them.
Heya,
Playing around with the 0.8.2-beta producer client. One of my test cases is
to ensure producers can deal with Kafka being down when the producer is
created. My tests failed miserably because of the default blocking in the
producer with regard to metadata.fetch.timeout.ms. The first line
Hi All,
Please give some inputs as this is pending since long and need to meet the
deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chha...@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: u...@storm.apache.org; users@kafka.apache.org
Subject: Issues With Parallelism
HI All -- has any one faced the problem i have posted in my previous email
?
Regards,
Nitin Kumar Sharma.
On Wed, Dec 17, 2014 at 11:08 AM, nitin sharma
wrote:
>
> Hi All,
>
> I am trying to figure out best configuration for my Kafka brokers so that
> in case of restarted, the new node catch u
Hello,
Playing around with the 0.8.2-beta producer client. One of my test cases
is to ensure producers can deal with Kafka being down when the producer is
created. My tests failed miserably because of the default blocking in the
producer with regard to metadata.fetch.timeout.ms. The first line o
Hey Paul,
Here are the constraints:
1. We wanted the storage of messages to be in their compact binary form so
we could bound memory usage. This implies partitioning prior to enqueue.
And as you note partitioning requires having metadata (even stale metadata)
about topics.
2. We wanted to avoid pr
@Jun, We can increase the number of resends, but the produce request may
still fail.
For async producer, at the time when it fails, we have
- messages that are in queue but has not been sent. From javaapi, we
don't know which messages are still in queue.
- Is it possible that we expose
Hello,
I am looking at 0.8.1.1, the kafka.producer.async.DefaultEventHandler
file. Below is the dispatchSerializedData function. Looks like we are
catching exception outside the loop and purely logs an error message.
We then return failedProduceRequests.
In case one broker is having problem, mess
Hello,
I am looking at 0.8.1.1, the kafka.producer.async.DefaultEventHandler
file. Below is the dispatchSerializedData function. Looks like we are
catching exception outside the loop and purely logs an error message.
We then return failedProduceRequests.
In case one broker is having problem, mess
I notice that there is parameter max.in.flight.requests.per.connection in the
ProducerConfig.java code that can be set. It is defaulted to 5.
This is not documented in the 0.8.2 documentation.
http://kafka.apache.org/082/documentation.html#newproducerconfigs
Is it something we should play with?
Hi David,
Each request sent to Kafka gets acknowledged. The protocol allows multiple
requests to be sent on a connection without waiting on a connection. The
number of requests currently awaiting acknowledgement is the in flight
request count. By default once there are five unacknowledged requests
Ray,
Thank you for the detailed explanation. I mistakenly thought this determined
the number of messages that were outstanding - not the number of requests
(batch of messages) that could be outstanding.
This makes sense.
Thank you.
-Original Message-
From: Jay Kreps [mailto:j...@confl
Dear all
We just build a kafka production cluster, I can create topics in kafka
production from another host. But when I am send very simple message as
producer, it generate such errors:
root@precise64:/etc/kafka# bin/kafka-console-producer.sh --broker-list
10.100.98.100:9092 --topic my-replicate
Looks like you can't connect to: 10.100.98.100:9092
I'd validate that this is the issue using telnet and then check the
firewall / ipfilters settings.
On Thu, Dec 18, 2014 at 2:21 PM, Sa Li wrote:
> Dear all
>
> We just build a kafka production cluster, I can create topics in kafka
> production
I would suggest to use the new client java producer in 0.8.2-beta
http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
it handles the case you brought up (among lots of other goodies).
/***
Joe Stein
Founder, Principal C
Thanks, Gwen, I telnet it,
root@precise64:/etc/kafka# telnet 10.100.98.100 9092
Trying 10.100.98.100...
Connected to 10.100.98.100.
Escape character is '^]'.
seems it connected, and I check with system operation people, netstate
should 9092 is listening. I am assuming this is the connection issue,
Perhaps you have the logs from broker? It may show other errors that
can help us troubleshoot.
On Thu, Dec 18, 2014 at 4:11 PM, Sa Li wrote:
> Thanks, Gwen, I telnet it,
> root@precise64:/etc/kafka# telnet 10.100.98.100 9092
> Trying 10.100.98.100...
> Connected to 10.100.98.100.
> Escape charact
The producer is complaining that it's socket channel is already closed.
Which makes me think it was closed due to some error that is not present in
your logs. I'd enable DEBUG and see if that shows the cause.
On Thu, Dec 18, 2014 at 4:13 PM, Gwen Shapira wrote:
>
> Perhaps you have the logs from
18 matches
Mail list logo
