I have created the JIRA: https://issues.apache.org/jira/browse/KAFKA-3726
–
Best regards,
Radek Gruchalski
ra...@gruchalski.com
de.linkedin.com/in/radgruchalski
Confidentiality:
This communication is intended for the above-named person and may be
confidential and/or legally privileged.
If it
Hi Tom,
There is, indeed, the problem with replication in case of the leader change for
the partition.
Hence, I think the best, best approach would to have Kafka emit events in case
of:
- partition leader change
- offset file to be cleaned up
This still leaves a lot of work for the ops people
The issue I have with that log.cleanup.policy approach is how it impacts
with replication. Different replicas often have different sets of segment
files. There are effectively two options then:
1. All replicas upload all their segments. This has issues with storage
space, but is easy to implement.
So Kafka is a fine solution as part of an event-sourced story. It's not a
simple solution, but it fits.
Kakfa can store data for a long time and you shouldn't discount this,
however, using it as the primary long-term data store might not be a good
fit if we're talking storing raw events for years
Dave:
Thanks for the suggestion, Bookkeeper looks interesting. I assume you meant
"more high-level interface"?
Chris:
Thanks Chris, I actually came across a few of your talks and articles when
researching this style of architecture (along with stuff by Greg Young and
Martin Kleppmann) and really
A curious question:
How does database + Kafka broker solution work?
Would the consumer(s) load the database and then overlay the events?
Or is it that the events are overlaid on the database and the database is
queried?
I am curious on how a complete solution would look like with "database +
Kafka
Oli,
Kafka is only a partial solution.
As I describe here (
http://www.slideshare.net/chris.e.richardson/hacksummit-2016-eventdriven-microservices-events-on-the-outside-on-the-inside-and-at-the-core/57)
an event store is a hybrid of a database and a message broker.
It is a database because it prov
wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> +1 to your solution of log.cleanup.policy. Other brokers (ie, ActiveMQ)
>>> have a feature like that.
>>> Is there a JIRA for this?
>
t; On Tue, May 17, 2016 at 4:48 PM, Radoslaw Gruchalski
> > wrote:
> >
> > > I have described a cold storage solution for Kafka:
> > >
> >
> https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv
> > .
> > > Also desc
.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv
> .
> > Also described it here a couple of times. Thd potential solution seems
> > rather straightforward.
> > Get Outlook for iOS
> >
> > _
> > From: Luke Steen
her straightforward.
> > Get Outlook for iOS
> >
> > _
> > From: Luke Steensen
> > Sent: Tuesday, May 17, 2016 11:22 pm
> > Subject: Re: Kafka for event sourcing architecture
> > To:
> >
> >
> > It's harder
_
> From: Luke Steensen
> Sent: Tuesday, May 17, 2016 11:22 pm
> Subject: Re: Kafka for event sourcing architecture
> To:
>
>
> It's harder in Kafka because the unit of replication is an entire
> partition, not a single key/value pair. Pa
Steensen
Sent: Tuesday, May 17, 2016 11:22 pm
Subject: Re: Kafka for event sourcing architecture
To:
It's harder in Kafka because the unit of replication is an entire
partition, not a single key/value pair. Partitions are large and constantly
growing, where key/value pairs are typically
a-cold-storage-32929d0a57b2#.kf0jf8cwv.
> Also described it here a couple of times. Thd potential solution seems
> rather straightforward.
> Get Outlook for iOS
>
> _
> From: Luke Steensen
> Sent: Tuesday, May 17, 2016 11:22 pm
> Subject
It's harder in Kafka because the unit of replication is an entire
partition, not a single key/value pair. Partitions are large and constantly
growing, where key/value pairs are typically much smaller and don't change
in size. There would theoretically be no difference if you had one
partition per k
I'm not sure why Kafka at least in theory cannot be used for infinite
retention – any replicated database system would need to have a new node
ingest all the data from failed node from its replicas. Surely this is no
different in S3 itself. Why is this harder to do in Kafka than in other
systems? T
Hi Oli,
Inline.
On Tuesday, 17 May 2016, Olivier Lalonde wrote:
> Hi all,
>
> I am considering adopting an "event sourcing" architecture for a system I
> am developing and Kafka seems like a good choice of store for events.
>
> For those who aren't aware, this architecture style consists in sto
Hi all,
I am considering adopting an "event sourcing" architecture for a system I
am developing and Kafka seems like a good choice of store for events.
For those who aren't aware, this architecture style consists in storing all
state changes of the system as an ordered log of events and building
18 matches
Mail list logo
