> they don't. So it sounds like dependencies may be an issue.
>
> Planning to give this more thought :)
>
> Gwen
>
> On Fri, May 1, 2015 at 7:56 PM, Jonathan Hodges wrote:
>
> > Hi Gwen,
> >
> > As you said I see Bottled Water and Sqoop managing s
Hi Gwen,
As you said I see Bottled Water and Sqoop managing slightly different use
cases so I don't see this feature as a Sqoop killer. However I did have a
question on your comment that the transaction log or CDC approach will have
problems with very large, very active databases.
I get that you
art again.
> 2) The messages that I last published, to make sure that I don't
> re-publish them.
>
> That does mean that all data from all tables is in a single topic. I will
> probably have a consumer that will read that "all tables" topic, and split
> the data o
Yes you are right on the oplog per partition as well as that mapping well
to the Kafka partitions. I think we are making this harder than it is
based on previous attempts and trying to leverage something like Databus
for propagating log changes from MongoDB and Cassandra since it requires a
scn.
ts should be met with minimal
> > impact
> > > > on
> > > > > message throughput or transaction rates within stores at either
> end.
> > > > >
> > > > > Let me know if this is more than what you (Jonathan) intended.
&
Sorry didn't realize the mailing list wasn't copied...
-- Forwarded message ------
From: Jonathan Hodges
Date: Wed, Jun 4, 2014 at 10:56 AM
Subject: Re: Hadoop Summit Meetups
To: Neha Narkhede
We have a number of customer facing online learning applications. These
ap
Hi Neha,
Thanks so much to you and the Kafka team for putting together the meetup.
It was very nice and gave people from out of town like us the ability to
join in person.
We are the guys from Pearson Education and we talked a little about
supplying some details on some of our use cases with res
We are also using straight Java. I am not sure if Spring/Camel supports
past 0.8 beta.
On Mon, Apr 14, 2014 at 9:39 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:
> In our case it's straight Java.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elas
Thanks so much, Neha!
On Fri, Apr 11, 2014 at 10:31 AM, Neha Narkhede wrote:
> Yes, that's a great idea. I can help organize the meetup at LinkedIn.
>
> Thanks,
> Neha
>
>
> On Fri, Apr 11, 2014 at 8:44 AM, Saurabh Agarwal (BLOOMBERG/ 731 LEXIN) <
> sagarwal...@bloomberg.net> wrote:
>
> > great
With the Hadoop Summit in San Jose 6/3 - 6/5 I wondered if any of the
LinkedIn geniuses were thinking of putting together a meet-up on any of the
associated technologies like Kafka, Samza, Databus, etc. For us poor souls
that don't live on the West Coast it was a great experience attending the
Kaf
+1 for security branch
We are willing to assist with the merge.
On Wed, Apr 2, 2014 at 8:32 AM, Joe Stein wrote:
> Hi Raja, do you have an ICLA https://www.apache.org/licenses/icla.txt on
> file with Apache?
>
> One thought would be to branch a security branch at the commit you forked
> from.
A cheaper approach is to use ephemeral disk and have the replicas across
AZs. You can lose up to 2 AZs and still avoid data loss. You mitigate
complete region failure using mirror maker to replicate to another region.
On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole wrote:
> Yeah, we use provis
; > Hi Jonathan
> >
> > We forked kafka to add SSL feature. It not part of kafka official release
> >
> > Sent from my iPhone
> >
> > On Nov 15, 2013, at 12:32 PM, Jonathan Hodges wrote:
> >
> > > Hi,
> > >
> > > While search
Hi,
While searching the user group messages I found the following thread -
http://grokbase.com/t/kafka/users/138vqq1x07/getting-leadernotavailableexception-in-console-producer-after-increasing-partitions-from-4-to-16.
It shows the following stack trace with 0.8.
[2013-08-27 08:29:30,372] INFO Fe
First off, I want to say this is awesome! It has been great to see all the
great YARN offerings being released lately. I noticed Hadoop 2.x was
recently voted beta so very exciting!
Like many we use Storm for near real-time processing our Kafka based
streams. In addition we send this data to Ha
;
> Thanks,
> Neha
>
>
> On Fri, Jul 26, 2013 at 7:31 AM, Jonathan Hodges
> wrote:
>
> > Hi LinkedIn Guys and Girls,
> >
> > Out of curiosity do you guys use any other messaging systems within
> > LinkedIn besides Kafka? For instance, are there any more
Hi LinkedIn Guys and Girls,
Out of curiosity do you guys use any other messaging systems within
LinkedIn besides Kafka? For instance, are there any more complex messaging
use cases requiring something like JMS or AMQ?
We recently replaced a JMS solution in favor of Kafka for a particular use
cas
is '^]'.
^CConnection closed by foreign host.
DNVCOML-2D3FFT3:~ uhodgjo$
On Tue, Jun 25, 2013 at 4:57 AM, Jonathan Hodges wrote:
> Hi Florin,
>
> I work with Yogesh so it is interesting you mention the
> 'metadata.broker.list' property as this was the first error
Hi Florin,
I work with Yogesh so it is interesting you mention the
'metadata.broker.list' property as this was the first error message we saw.
Consider the following producer code.
Properties props = new Properties();
props.put("broker.list", "x.x.x.x:9092, x.x.x.x :9092, x.x.x.x :9092,
x.x.x.x
, Jun Rao wrote:
> If anyone is interested in presenting their use case at our user group
> meeting, please let use know. We can add it to the agenda.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jun 20, 2013 at 8:07 AM, Jonathan Hodges
> wrote:
>
> > Definitely interes
?
>
> Thanks,
>
> Jun
>
>
> On Thu, Jun 20, 2013 at 4:41 AM, Jonathan Hodges
> wrote:
>
> > Hi Neha,
> >
> > Any update on the Thursday evening meeting venue?
> >
> > Thanks,
> > Jonathan
> >
> >
> >
> > On Tue,
t; > > Thanks,
> > > Neha
> > >
> > >
> > > On Tue, May 21, 2013 at 10:54 AM, Vaibhav Puranik > > >wrote:
> > >
> > > > Any day will work for me. I am waiting for you to decide on time so
> > that
> > > I
> >
is incorrect.
> >
> > Thanks!
> >
> >
> > On Tue, Jun 11, 2013 at 8:20 AM, Tim Watson >wrote:
> >
> >> Hi Jonathan,
> >>
> >> So, thanks for replying - that's all useful info.
> >>
> >> On 10 Jun 2013, at 1
Thanks!
On Thu, Jun 13, 2013 at 8:33 AM, Neha Narkhede wrote:
> No. It only means that messages are written to all replicas in memory. Data
> is flushed to disk asynchronously.
>
> Thanks,
> Neha
> On Jun 13, 2013 3:29 AM, "Jonathan Hodges" wrote:
>
> &
Looking at Jun’s ApacheCon slides (
http://www.slideshare.net/junrao/kafka-replication-apachecon2013) slide 21
titled, ‘Data Flow in Replication’ there are three possible durability
configurations which tradeoff latency for greater persistence guarantees.
The third row is the ‘no data loss’ config
.
Thanks!
On Tue, Jun 11, 2013 at 8:20 AM, Tim Watson wrote:
> Hi Jonathan,
>
> So, thanks for replying - that's all useful info.
>
> On 10 Jun 2013, at 14:19, Jonathan Hodges wrote:
> > Kafka has a configurable rolling window of time it keeps the messages per
> > topic.
get much closer to exactly once guarantees i.e. the consumers can manage
their consumption state as well as the consumed messages in the same
transaction.
On Mon, Jun 10, 2013 at 6:40 AM, Tim Watson wrote:
> Hi Jonathan,
>
> Cheers,
> Tim
>
> On 10 Jun 2013, at 13:12, Jonathan H
> What if you messaging requirements are in the 100's GBSs? Would you say
> RabbitMQ is probably a better fit?
>
> On Jun 8, 2013, at 4:03 PM, Jonathan Hodges wrote:
>
> > I am not making any assumptions other than Rabbit needs to maintain the
> > state of the consu
es and other consumers within the same group. For a given
> topic and a given consumer group, broker partitions are divided evenly
> among consumers within the group."
>
> When a new consumer and/or partition appears, can messages in the
> broker get "moved" from one partitio
On Sat, Jun 8, 2013 at 2:09 AM, Jonathan Hodges wrote:
> Thanks so much for your replies. This has been a great help understanding
> Rabbit better with having very little experience with it. I have a few
> follow up comments below.
Happy to help!
I'm afraid I don't follow y
2013 at 4:40 PM, Alexis Richardson <
alexis.richard...@gmail.com> wrote:
> Jonathan,
>
>
> On Fri, Jun 7, 2013 at 7:03 PM, Jonathan Hodges wrote:
> > Hi Alexis,
> >
> > I appreciate your reply and clarifications to my misconception about
> > Rabbit, particular
abbit if you wanted to share the same stream of messages across
multiple consumers.
On Fri, Jun 7, 2013 at 12:03 PM, Jonathan Hodges wrote:
> Hi Alexis,
>
> I appreciate your reply and clarifications to my misconception about
> Rabbit, particularly on the copying of the message payl
s, it will get
> duplicated
> > in the queue of each of those consumer. Kafka wins on that side too since
> > every consumer reads from the same source.
> >
> > The downsides of Kafka were the language issues (we are using mostly
> Python
> > and C#). 0.8 is ver
We just went through a similar exercise with RabbitMQ at our company with
streaming activity data from our various web properties. Our use case
requires consumption of this stream by many heterogeneous consumers
including batch (Hadoop) and real-time (Storm). We pointed out that Kafka
acts as a c
Awesome right up Jason! Very helpful as we are also looking to build a
Kafka environment in AWS. I am curious, are you using Kafka 0.7.2 or 0.8
in your tests? Did you have just one EBS volume per broker instance or
RAID 10 across EBS volumes per broker?
Thanks again for the great info!
-Jonath
Tuesday or Thursday works the best for me as I plan to attend the Summit
Party on Wednesday evening. Thanks!
On Mon, May 20, 2013 at 10:05 PM, Jun Rao wrote:
> Yes, we can have a Kafka user group meeting then. We could do this in one
> of the evenings (Tue, Wed, or Thu). What will people prefe
Great idea Vaibhav! I would also be interested in this as I live in Denver
and don't get to the Bay area too often.
-Jonathan
On Mon, May 20, 2013 at 2:35 PM, Vaibhav Puranik wrote:
> Jun and Neha,
>
> Is there any plan for Kafka Users group meeting around Hadoop Summit?
>
> It was done last
er API post 0.8 to expose this information.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 27, 2013 at 2:41 PM, Jonathan Hodges
> wrote:
>
> > I know this is a really old thread, but it looked like the only pertinent
> > one that came up when searching for ‘exactly once’ in the a
I know this is a really old thread, but it looked like the only pertinent
one that came up when searching for ‘exactly once’ in the archives. I just
want to confirm my understanding of the 0.8 version in that it still
doesn’t completely support exactly once semantics. With the producer
configured
Many thanks for contributing! The docs are very helpful. I found a couple
small possible typos. The partitioning code example looks like it repeats
at the bottom with duplicate import and class definition statements. Also
the create topic command-line appears to have an extra '-' for the
partit
40 matches
Mail list logo
