I'm curious what type of batching Kafka producers do at the socket layer.
For instance, if I have a partitioner that round robin's n messages to a
different partition, am I guaranteed to get n different messages sent over
the socket or is there some micro-batching going on underneath?
I am trying
I'm new to Kafka and I'm trying to understand the version semantics. We
want to use Kafka w/ Spark but our version of Spark is tied to 0.8.0. We
were wondering what guarantees are made about backwards compatbility across
0.8.x.x. At first glance, given the 3 digits used for versions, I figured
0.8.
batching in new producer is
> per topic partition, the batch size it is controlled by both max batch
> size and linger time config.
>
> Jiangjie (Becket) Qin
>
> On 3/9/15, 10:10 AM, "Corey Nolet" wrote:
>
> >I'm curious what type of batching Kafka prod
I have a cluster of 3 nodes and I've created a topic with some number of
partitions and some number of replicas, let's say 10 and 2, respectively.
Later, after I've got my 3 nodes fairly consumed with data in the 10
partitions, I want to add 2 more nodes to the mix to help balance out the
partition
I'm firing up a KafkaServer (using some EmbeddedKafkaBroker code that I
found on Github) so that I can run an end-to-end test ingesting data
through a kafka topic with consumers in Spark Streaming pushing to
Accumulo.
Thus far, my code is doing this:
1) Creating a MiniAccumuloCluster and KafkaSer
t it needs to add the partition assignment to
zookeeper with itself as the leader but it's strange that the log messages
above seem like they are missing the data. "New topic creation callback for
" seems like it should be listing a topic and not blank.
Any ideas?
On Thu, May 14, 2015
Json encoded blob definitely appears to be going in as a json string. The
partition assignment json seems to be the only thing that is being prefixed
by these bytes. Any ideas?
On Thu, May 14, 2015 at 5:17 PM, Corey Nolet wrote:
> I think I figured out what the problem is, though I'm
beginning of it that are making it
unparseable once pulled from zookeeper.
Any ideas to what this could be? I'm using 0.8.2.0- this is really what's
holding me back right now from getting my tests functional.
On Thu, May 14, 2015 at 4:29 PM, Corey Nolet wrote:
> I raised the log le
