Assuming that you use a single producer in the async mode, the Kafka overhead should be limited to a single thread. Using a cheaper compression codec such as snappy will also help reducing the CPU load.
Thanks, Jun On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj <reach.krishna...@gmail.com> wrote: > Hi Jun, > > Thanks for replying back on this. Appreciated. > > I do understand that the Kafka Client just needs a protocol compatibility > with the Application which is producing the messages. > > To clarity a bit more: > > I witnessed a scenario where a large scale website uses the Kafka Library > in their Web Application. So in this case, the Kafka libraries are tied to > the Application which are served by Web Servers. > > So, When there was an issue caused by Kafka related to CPU usage, the team > wanted to do a patch. In this case, in order to do a patch, they had to > create a new WAR package and deploy again in Web Server which is a > significant effort. > > I totally understand that having a layer like Logging service in between > Kafka and the Application will totally defect the purpose for Kafka. > > And I would love to know your advice how best to handle these type of > maintenance. > > Thanks, > Krishna Raj > > > > > > On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <jun...@gmail.com> wrote: > >> Could you be a bit more specific about the issue? As long as there is >> protocol compatibility btw the Kafka client and the broker, upgrading the >> Kafka client library should be easy, right? >> >> Thanks, >> >> Jun >> >> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <reach.krishna...@gmail.com> >> wrote: >> >>> Hello Amazing Kafka Creators & User, >>> >>> I have learnt and use kafka in our Production system, so you can count >>> my understanding as intermediate. >>> >>> With the statement that "Kafka has solved the Scalability and >>> Availability needs for a large scale message publish/subscribe system", I >>> understand that having a Producer Service which sits in between the >>> Application and the Producer defects the one major purpose of Kafka. >>> >>> So, my question is, How to loosely couple Kafka with my Production >>> Application ? >>> >>> The reason being, I wish to do all producer code and Kafka library >>> maintenance without affecting my large scale Production system. Its not an >>> easy thing to buy a window to these type of changes done on a large scale >>> production application :) >>> >>> Any advice on how this can be achieved(even moderately) will greatly >>> help ? >>> >>> Thanks, >>> Krishna Raj >>> >> >> >
