This would be of value to me, as well. I'm currently not sure how to avoid
having users of ruby-kafka produce messages that exceed that limit when
using an async producer loop – I'd prefer to not allow such a message into
the buffers at all rather than having to deal with it only when there's a
bro
x27;t accept an offset for a
message that has expired.
Do you have any experience in building a system such as this?
Daniel Schierbeck
Senior Staff Engineer @ Zendesk
I'm not sure why Kafka at least in theory cannot be used for infinite
retention – any replicated database system would need to have a new node
ingest all the data from failed node from its replicas. Surely this is no
different in S3 itself. Why is this harder to do in Kafka than in other
systems? T
Have you looked into using a relational database as the primary store, with
something like Maxwell or Bottled Water as a broadcast mechanism?
On Mon, 28 Mar 2016 at 17:28 Daniel Schierbeck wrote:
> I ended up abandoning the use of Kafka as a primary event store, for
> several reasons. One
I ended up abandoning the use of Kafka as a primary event store, for
several reasons. One is the partition granularity issue; another is the
lack of a way to guarantee exclusive write access, i.e. ensure that only a
single process can commit an event for an aggregate at any one time.
On Mon, 28 Mar
Since Kafka itself has replication, I'm not sure what HDFS backups would
bring – how would you recover from e.g. all Kafka nodes blowing up if you
only have an HDFS backup? Why not use MirrorMaker to replicate the cluster
to a remote DC, with a process of reversing the direction in case you need
to
a.guru%3E>>.
> So in theory one
> could grow a single partition to terabytes-scale. But don’t take my word
> for it, as I have not tried it.
>
> Cheers, Giidox
>
>
>
> > On 09 Mar 2016, at 15:10, Daniel Schierbeck
> wrote:
> >
> > I'm co
deas they're been pushing. I'd also
like to know what sort of problems Kafka would pose for long-term storage –
would I need special storage nodes, or would replication be sufficient to
ensure durability?
Daniel Schierbeck
Senior Staff Engineer, Zendesk
I'm also very interested in using Kafka as a persistent, distributed commit
log – essentially the write side of a distributed database, with the read
side being an array of various query stores (Elasticsearch, Redis,
whatever) and stream processing systems.
The benefit of retaining data in Kafka i
va clients, but many more are also using
> non-Java clients and it's great to see more clients supported across many
> languages. Even more compelling to see libraries deployed in production!
>
> -Ewen
>
>
> On Wed, Feb 3, 2016 at 7:11 AM, Daniel Schierbeck
> > wrote:
>
tly running in production at Zendesk, handling about
1,000 messages/s across our data centers.
Best regards,
Daniel Schierbeck
onsumer requests the message.
Does anyone have experience with this, or do you just let the Kafka topic
delete old messages? I'd much prefer keeping the data in Kafka forever, as
it's ideally suited for bootstrapping new systems, e.g. search indexes,
analytics, etc.
Best regards,
Daniel Schierbeck
ing --
> just for the partitionwide version at the moment, since it turns out
> to be pretty simple to implement. Still very interested in moving
> forward with this stuff, though not always as much time as I would
> like...
>
> On Thu, Jul 9, 2015 at 9:39 AM, Daniel Schierbeck
&
.
Daniel Schierbeck
> On 13. jul. 2015, at 15.41, Scott Thibault
> wrote:
>
> We've tried to use Kafka not as a persistent store, but as a long-term
> archival store. An outstanding issue we've had with that is that the
> broker holds on to an open file handle on
Would it be possible to document how to configure Kafka to never delete
messages in a topic? It took a good while to figure this out, and I see it
as an important use case for Kafka.
On Sun, Jul 12, 2015 at 3:02 PM Daniel Schierbeck <
daniel.schierb...@gmail.com> wrote:
>
> > O
> On 10. jul. 2015, at 23.03, Jay Kreps wrote:
>
> If I recall correctly, setting log.retention.ms and log.retention.bytes to
> -1 disables both.
Thanks!
>
> On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck <
> daniel.schierb...@gmail.com> wrote:
>
>
ra...@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you
25-everything-about-kafka-part-2/
>
> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck <
> daniel.schierb...@gmail.com> wrote:
>
>> I'd like to use Kafka as a persistent store – sort of as an alternative to
>> HDFS. The idea is that I'd load the data
I'd like to use Kafka as a persistent store – sort of as an alternative to
HDFS. The idea is that I'd load the data into various other systems in
order to solve specific needs such as full-text search, analytics, indexing
by various attributes, etc. I'd like to keep a single source of truth,
howeve
Ben, are you still interested in working on this?
On Mon, Jun 15, 2015 at 9:49 AM Daniel Schierbeck <
daniel.schierb...@gmail.com> wrote:
> I like to refer to it as "conditional write" or "conditional request",
> semantically similar to HTTP's If-Match header
current record for the
> > key, then yes. This might work fine for applications that track key,
> value,
> > and offset. It is not quite the same as doing a normal CAS.
> >
> > On Sat, Jun 13, 2015 at 12:07 PM, Daniel Schierbeck <
> > daniel.schierb...@gmail.com&g
ce random disk seeks on any CAS operation (and making page caching
> harder for the OS, etc.).
>
>
> On Sat, Jun 13, 2015 at 11:33 AM, Daniel Schierbeck <
> daniel.schierb...@gmail.com> wrote:
>
> > Ewen: would single-key CAS necessitate random reads? My idea was to hav
> performance costs & data structures are isolated to background threads.
>
> -Ewen
>
> On Sat, Jun 13, 2015 at 9:59 AM, Daniel Schierbeck <
> daniel.schierb...@gmail.com> wrote:
>
> > @Jay:
> >
> > Regarding your first proposal: wouldn't that mea
the implementation seems a bit
> > more involved. Want to add it to the ticket?
> >
> > On Fri, Jun 12, 2015 at 7:49 AM, Daniel Schierbeck
> > wrote:
> > > Ben: your solutions seems to focus on partition-wide CAS. Have you
> > > considered per-key CAS? That w
ght; the stream consumer would be
> > responsible for finalizing one booking, and notifying the other client
> > that their request had failed. (In-browser or by email.)
> >
> > On Wed, Jun 10, 2015 at 5:04 AM, Daniel Schierbeck
> > wrote:
> >> I've bee
I've been working on an application which uses Event Sourcing, and I'd like
to use Kafka as opposed to, say, a SQL database to store events. This would
allow me to easily integrate other systems by having them read off the
Kafka topics.
I do have one concern, though: the consistency of the data ca
26 matches
Mail list logo
