+1 to making the API use bytes and push serialization into the client. This
is effectively what I am doing currently anyway. I implemented a generic
Encoder which just passes the bytes through.
I also like the idea of the client being written in pure Java. Interacting
with Scala code from Java isn
Jay,
This seems like a great direction. Simplifying the consumer client would
be a big win, and +1 for more native java client integration.
On the last point, regarding memory usage for buffering per partition. I
would think it could be possible to devise a dynamic queuing system, to
allow high
Cool, good to know.
On Fri, Jul 26, 2013 at 2:00 PM, Jason Rosenberg wrote:
> Jay,
>
> My only experience so far with this is using XFS. It appears the XFS
> behavior is evolving, and in fact, we see somewhat different behavior from
> 2 of our CentOS kernel versions in use. I've been trying t
Jay,
My only experience so far with this is using XFS. It appears the XFS
behavior is evolving, and in fact, we see somewhat different behavior from
2 of our CentOS kernel versions in use. I've been trying to ask questions
about all this on the XFS.org mailing list, but so far, having not much
l
I sent around a wiki a few weeks back proposing a set of client
improvements that essentially amount to a rewrite of the producer and
consumer java clients.
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
The below discussion assumes you have read this wiki.
I started to do a li
Interesting.
Yes, Kafka keeps all log files open indefinitely. There is no inherent
reason this needs to be the case, though, it would be possible to LRU out
old file descriptors and close them if they are not accessed for a few
hours and then reopen on the first access. We just haven't implemente
This NOT a harmless race.
Now my QA teammate is encountering this issue under load. The result of it is a
background thread that is spinning in a loop that always hits a
NullPointerException.
I have implemented a variety of assurances in my application code to ensure
that the high-level consum
Hey guys,
We decided to use Kafka in our new project, now I spend some time to
research how Kafka producer behaves while network connectivity
problems.
I had 3 virtual machines(ubuntu 13.04, running on Virtualbox) in one network:
1. Kafka server(0.7.2) + Zookeper.
2. Producer app with default se
Awesome! This is very helpful Neha.
On Fri, Jul 26, 2013 at 9:29 AM, Neha Narkhede wrote:
> Jonathan,
>
> We are in the process of replacing our legacy ActiveMQ based applications
> with Kafka. As such all messaging, logging and queuing applications use
> Kafka at LinkedIn.
>
> Thanks,
> Neha
>
Jonathan,
We are in the process of replacing our legacy ActiveMQ based applications
with Kafka. As such all messaging, logging and queuing applications use
Kafka at LinkedIn.
Thanks,
Neha
On Fri, Jul 26, 2013 at 7:31 AM, Jonathan Hodges wrote:
> Hi LinkedIn Guys and Girls,
>
> Out of curiosit
Hi LinkedIn Guys and Girls,
Out of curiosity do you guys use any other messaging systems within
LinkedIn besides Kafka? For instance, are there any more complex messaging
use cases requiring something like JMS or AMQ?
We recently replaced a JMS solution in favor of Kafka for a particular use
cas
It looks like xfs will reclaim the preallocated space for a file, after it
is closed.
Does kafka close a file after it has reached it's max size and started
writing to the next log file in sequence? Or does it keep all open until
they are deleted, or the server quits (that's what it seems like).
12 matches
Mail list logo
