> We are currently using spark streaming 1.2.1 with kafka and write-ahead log.
> I will only say one thing : "a nightmare". ;-)
I’d be really interested in hearing about your experience here.  I’m exploring 
streaming frameworks a bit, and Spark Streaming is just so easy to use and set 
up.  I’d be nice if it worked well.




> On Mar 13, 2015, at 15:38, Alberto Miorin <amiorin78+ka...@gmail.com> wrote:
> 
> We are currently using spark streaming 1.2.1 with kafka and write-ahead log.
> I will only say one thing : "a nightmare". ;-)
> 
> Let's see if things are better with 1.3.0 :
> http://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html
> 
> On Fri, Mar 13, 2015 at 8:33 PM, William Briggs <wrbri...@gmail.com> wrote:
> 
>> Spark Streaming also has built-in support for Kafka, and as of Spark 1.2,
>> it supports using an HDFS write-ahead log to ensure zero data loss while
>> streaming:
>> https://databricks.com/blog/2015/01/15/improved-driver-fault-tolerance-and-zero-data-loss-in-spark-streaming.html
>> 
>> -Will
>> 
>> On Fri, Mar 13, 2015 at 3:28 PM, Alberto Miorin <amiorin78+ka...@gmail.com
>>> wrote:
>> 
>>> I'll try this too. It looks very promising.
>>> 
>>> Thx
>>> 
>>> On Fri, Mar 13, 2015 at 8:25 PM, Gwen Shapira <gshap...@cloudera.com>
>>> wrote:
>>> 
>>>> There's a KafkaRDD that can be used in Spark:
>>>> https://github.com/tresata/spark-kafka. It doesn't exactly replace
>>>> Camus, but should be useful in building Camus-like system in Spark.
>>>> 
>>>> On Fri, Mar 13, 2015 at 12:15 PM, Alberto Miorin
>>>> <amiorin78+ka...@gmail.com> wrote:
>>>>> We use spark on mesos. I don't want to partition our cluster because
>>> of
>>>> one
>>>>> YARN job (camus).
>>>>> 
>>>>> Best
>>>>> 
>>>>> Alberto
>>>>> 
>>>>> On Fri, Mar 13, 2015 at 7:43 PM, Otis Gospodnetic <
>>>>> otis.gospodne...@gmail.com> wrote:
>>>>> 
>>>>>> Just curious - why - is Camus not suitable/working?
>>>>>> 
>>>>>> Thanks,
>>>>>> Otis
>>>>>> --
>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
>>> Management
>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>> 
>>>>>> 
>>>>>> On Fri, Mar 13, 2015 at 2:33 PM, Alberto Miorin <
>>>> amiorin78+ka...@gmail.com
>>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> I was wondering if anybody has already tried to mirror a kafka
>>> topic
>>>> to
>>>>>>> hdfs just copying the log files from the topic directory of the
>>> broker
>>>>>>> (like 00000000000023244237.log).
>>>>>>> 
>>>>>>> The file format is very simple :
>>>>>>> https://twitter.com/amiorin/status/576448691139121152/photo/1
>>>>>>> 
>>>>>>> Implementing an InputFormat should not be so difficult.
>>>>>>> 
>>>>>>> Any drawbacks?
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 
>> 

Reply via email to