Just a followup for completeness sake -- for testing purpose I have setup fluentd and fluent-plugin-kafka plugin. Now my android APP now does POST to this fluent instance which forwards the message on to Kafka and then subsequent items in the processing pipeline.
Cheers, -Subodh On Wed, Jul 16, 2014 at 10:06 PM, Subodh Nijsure < subodh.nijs...@sigsensetech.com> wrote: > Thanks Philip, I will certainly take a look. > > I had not considered the fact that Kafka doesn't have any > security/authentication mechanism yet, so using some kind of secure gateway > certainly makes sense. > > -Subodh > > > On Wed, Jul 16, 2014 at 10:03 PM, Philip O'Toole < > philip_o_to...@yahoo.com.invalid> wrote: > >> You should find code here that will help you get a HTTP app server >> together, that writes to Kafka on the back-end. >> >> https://cwiki.apache.org/confluence/display/KAFKA/Clients >> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem >> >> >> >> >> On Wednesday, July 16, 2014 9:36 PM, Philip O'Toole >> <philip.oto...@yahoo.com.INVALID> wrote: >> >> >> >> FWIW, I happen to know Subodh -- we worked together many years back. We >> discussed this a little off-the-list, but perhaps my thoughts might be of >> wider interest. >> >> Kafka, in my experience, works best when Producers have a persistent TCP >> connection to the Broker(s) (and possibly Zookeeper). I guess that it isn't >> really practical for a mobile device. Also, Kafka messages are a specific >> protocol over TCP, which will need to be coded for the Android platform. I >> don't know how difficult this is, but doesn't seem worth it. >> >> Putting a lightweight HTTP server in front of Kafka is definitely the way >> to go. This could be written in Go, Python, Ruby, node.js -- almost >> anything. These servers will be stateless, and you can run many of them >> behind a load-balancer or, perhaps AWS Route 53, for redundancy. All >> standard stuff, and it's pretty easy nowadays to build scalable web >> servers. What happens to the data afterwards is much harder. :-) >> >> Furthermore, I'm sure it's much easier and quicker to write code on >> Android to perform an arbitrary POST to a URL in the Cloud. Mocking out an >> HTTP endpoint is very easy too, for the purposes of testing the client >> software. >> >> And the HTTP server can actually be HTTPS, giving the software at the >> edge confidence it is talking to the real endpoint. Perhaps run the HTTP >> servers behind nginx, and let it do the SSL termination. Secure transport >> is not built-into Kafka, last time I checked. >> >> So while strictly speaking it is more software, the system is >> significantly more manageable and robust. >> >> Philip >> >> >> On Wednesday, July 16, 2014 6:33 PM, Steve Robenalt < >> sroben...@highwire.org> wrote: >> >> >> >> Hi Subodh, >> >> I would think you'd be better off having an app server of some kind as an >> intermediary that accepts messages from your android app and posts them to >> Kafka for you, rather than having your app be a Kafka Producer on its own. >> >> Steve >> >> >> >> On Wed, Jul 16, 2014 at 1:22 PM, Subodh Nijsure < >> subodh.nijs...@sigsensetech.com> wrote: >> >> > Hello, >> > >> > I am trying to make my Android application to be the producer of >> messages >> > that get sent to Kafka server. >> > >> > I am trying to compile my android code using following libraries - >> > kafka_2.9.2-0.8.1.1.jar, scala-library-2.9.2.jar >> > >> > However I am running into issue where it produces too many methods. >> > >> > So the question: >> > >> > Has anyone been successful in sending messages form Android application >> to >> > server. If I want to trim the libraries should I be just compiling >> source >> > myself or is there and/or any other alternative I should look at to get >> my >> > android app to send message to Kafka server. >> > >> > -Subodh >> > >> > >
