To circle back around this thread there is an available native .net client https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net now.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Tue, May 13, 2014 at 5:01 AM, Margusja <mar...@roo.ee> wrote: > Ok got some info myself. > > I can have fault tolerance - I can start kafka-http-endpoint using broker > lists > I can have ack - start using --sync > > But what is best practice in case if kafka-http-endpoint goes down? > > Start multiple kafka-http-endpoint's and in client side just control that > kafka-http-endpoint is up? And if not up then using another? > > > Best regards, Margus (Margusja) Roo > +372 51 48 780 > http://margus.roo.ee > http://ee.linkedin.com/in/margusroo > skype: margusja > ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)" > > On 13/05/14 10:49, Margusja wrote: > >> Thank you for response. I think HTTP is ok. >> I have two more question in case of HTTP. >> 1. Can I have fault tolerance in case I have two or more brokers? >> 2. Can I ack that message is in queue? >> >> Best regards, Margus (Margusja) Roo >> +372 51 48 780 >> http://margus.roo.ee >> http://ee.linkedin.com/in/margusroo >> skype: margusja >> ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)" >> >> On 12/05/14 23:28, Joe Stein wrote: >> >>> The wire protocol has changed drastically since then. >>> >>> I don't know of any C# clients (there are none on the client library page >>> nor have I heard of any being used in production but maybe there are >>> some). >>> >>> >>> For clients that use DotNet I often suggest that they use some HTTP >>> producer/consumer >>> https://cwiki.apache.org/confluence/display/KAFKA/ >>> Clients#Clients-HTTPREST >>> >>> /******************************************* >>> Joe Stein >>> Founder, Principal Consultant >>> Big Data Open Source Security LLC >>> http://www.stealth.ly >>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >>> ********************************************/ >>> >>> >>> On Mon, May 12, 2014 at 12:34 PM, Margusja <mar...@roo.ee> wrote: >>> >>> Hi >>>> >>>> I have kafka broker running (kafka_2.9.1-0.8.1.1) >>>> All is working. >>>> >>>> One project requires producer is written in CSharp >>>> I am not dot net programmer but I managed to write simple producer code >>>> using https://github.com/kafka-dev/kafka/blob/master/clients/ >>>> csharp/README.md >>>> >>>> the code >>>> ... >>>> using System; >>>> using System.Collections.Generic; >>>> using System.Text; >>>> using System.Threading.Tasks; >>>> using Kafka.Client; >>>> >>>> namespace DemoProducer >>>> { >>>> class Program >>>> { >>>> static void Main(string[] args) >>>> { >>>> string payload1 = "kafka 1."; >>>> byte[] payloadData1 = Encoding.UTF8.GetBytes(payload1); >>>> Message msg1 = new Message(payloadData1); >>>> >>>> string payload2 = "kafka 2."; >>>> byte[] payloadData2 = Encoding.UTF8.GetBytes(payload2); >>>> Message msg2 = new Message(payloadData2); >>>> >>>> Producer producer = new Producer("broker", 9092); >>>> producer.Send("kafkademo3", 0 , msg1 ); >>>> } >>>> } >>>> } >>>> ... >>>> >>>> In broker side I am getting the error if I executing the code above: >>>> >>>> [2014-05-12 19:15:58,984] ERROR Closing socket for /84.50.21.39 because >>>> of error (kafka.network.Processor) >>>> java.nio.BufferUnderflowException >>>> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145) >>>> at java.nio.ByteBuffer.get(ByteBuffer.java:694) >>>> at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:38) >>>> at kafka.api.ProducerRequest$.readFrom(ProducerRequest. >>>> scala:33) >>>> at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys. >>>> scala:36) >>>> at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys. >>>> scala:36) >>>> at kafka.network.RequestChannel$Request.<init>(RequestChannel. >>>> scala:53) >>>> at kafka.network.Processor.read(SocketServer.scala:353) >>>> at kafka.network.Processor.run(SocketServer.scala:245) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> >>>> [2014-05-12 19:16:11,836] ERROR Closing socket for /90.190.106.56 >>>> because >>>> of error (kafka.network.Processor) >>>> java.io.IOException: Connection reset by peer >>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) >>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) >>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) >>>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) >>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java: >>>> 379) >>>> at kafka.utils.Utils$.read(Utils.scala:375) >>>> at kafka.network.BoundedByteBufferReceive.readFrom( >>>> BoundedByteBufferReceive.scala:54) >>>> at kafka.network.Processor.read(SocketServer.scala:347) >>>> at kafka.network.Processor.run(SocketServer.scala:245) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> I suspected that the problem is in the broker version >>>> (kafka_2.9.1-0.8.1.1) so I downloaded kafka-0.7.1-incubating. >>>> Now I was able to send messages using CSharp code. >>>> >>>> So is there workaround how I can use latest kafka version and CSharp ? >>>> Or >>>> What is the latest kafka version supporting CSharp producer? >>>> >>>> -- >>>> Best regards, Margus (Margusja) Roo >>>> +372 51 48 780 >>>> http://margus.roo.ee >>>> http://ee.linkedin.com/in/margusroo >>>> skype: margusja >>>> ldapsearch -x -h ldap.sk.ee -b c=EE "(serialNumber=37303140314)" >>>> >>>> >>>> >> >
