Ok. - so we figured out what the problem was with the consumers lagging
behind.
We were pushing 800Mbits/sec+ to the consumer interface - so the 1Gb
network interface was maxed out.
Graeme
On Wed, Oct 2, 2013 at 2:35 PM, Graeme Wallace <
graeme.wall...@farecompare.com> wrote:
> Yes, definit
, October 02, 2013 4:36 PM
To: users
Subject: Re: Strategies for improving Consumer throughput
Yes, definitely consumers are behind - we can see from examining the offsets
On Wed, Oct 2, 2013 at 1:59 PM, Joe Stein wrote:
> Are you sure the consumers are behind? could the pause be because
Yes, definitely consumers are behind - we can see from examining the offsets
On Wed, Oct 2, 2013 at 1:59 PM, Joe Stein wrote:
> Are you sure the consumers are behind? could the pause be because the
> stream is empty and producing messages is what is behind the consumption?
>
> What if you shut
This is with 0.8
On Wed, Oct 2, 2013 at 2:00 PM, Philip O'Toole wrote:
> Is this with 0.7 or 0.8?
>
>
> On Wed, Oct 2, 2013 at 12:59 PM, Joe Stein wrote:
>
> > Are you sure the consumers are behind? could the pause be because the
> > stream is empty and producing messages is what is behind the
Is this with 0.7 or 0.8?
On Wed, Oct 2, 2013 at 12:59 PM, Joe Stein wrote:
> Are you sure the consumers are behind? could the pause be because the
> stream is empty and producing messages is what is behind the consumption?
>
> What if you shut off your consumers for 5 minutes and then start the
Are you sure the consumers are behind? could the pause be because the
stream is empty and producing messages is what is behind the consumption?
What if you shut off your consumers for 5 minutes and then start them again
do the consumers behave the same way?
/**
Hi All,
We've got processes that produce many millions of itineraries per minute.
We would like to get them into HBase (so we can query for chunks of them
later) - so our idea was to write each itinerary as a message into Kafka -
so that not only can we have consumers that write to HBase, but also
