Ahh the problem probably is async ingestion to Spark receiver buffers,
hence WAL is required I would say.

Petr

On Tue, Sep 22, 2015 at 10:53 AM, Petr Novak <oss.mli...@gmail.com> wrote:

> If MQTT can be configured with long enough timeout for ACK and can buffer
> enough events while waiting for Spark Job restart then I think one could do
> even without WAL assuming that Spark job shutdowns gracefully. Possibly
> saving its own custom metadata somewhere, f.e. Zookeeper, if required to
> restart Spark job.
>
> On Tue, Sep 22, 2015 at 10:53 AM, Petr Novak <oss.mli...@gmail.com> wrote:
>
>> Ahh the problem probably is async ingestion to Spark receiver buffers,
>> hence WAL is required I would say.
>>
>> On Tue, Sep 22, 2015 at 10:52 AM, Petr Novak <oss.mli...@gmail.com>
>> wrote:
>>
>>> If MQTT can be configured with long enough timeout for ACK and can
>>> buffer enough events while waiting for Spark Job restart then I think one
>>> could do even without WAL assuming that Spark job shutdowns gracefully.
>>> Possibly saving its own custom metadata somewhere, f.e. Zookeeper, if
>>> required to restart Spark job.
>>>
>>> Petr
>>>
>>> On Mon, Sep 21, 2015 at 8:49 PM, Adrian Tanase <atan...@adobe.com>
>>> wrote:
>>>
>>>> I'm wondering, isn't this the canonical use case for WAL + reliable
>>>> receiver?
>>>>
>>>> As far as I know you can tune Mqtt server to wait for ack on messages
>>>> (qos level 2?).
>>>> With some support from the client libray you could achieve exactly once
>>>> semantics on the read side, if you ack message only after writing it to
>>>> WAL, correct?
>>>>
>>>> -adrian
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 21 Sep 2015, at 12:35, Petr Novak <oss.mli...@gmail.com> wrote:
>>>>
>>>> In short there is no direct support for it in Spark AFAIK. You will
>>>> either manage it in MQTT or have to add another layer of indirection -
>>>> either in-memory based (observable streams, in-mem db) or disk based
>>>> (Kafka, hdfs files, db) which will keep you unprocessed events.
>>>>
>>>> Now realizing, there is support for backpressure in v1.5.0 but I don't
>>>> know if it could be exploited aka I don't know if it is possible to
>>>> decouple event reading into memory and actual processing code in Spark
>>>> which could be swapped on the fly. Probably not without some custom built
>>>> facility for it.
>>>>
>>>> Petr
>>>>
>>>> On Mon, Sep 21, 2015 at 11:26 AM, Petr Novak <oss.mli...@gmail.com>
>>>> wrote:
>>>>
>>>>> I should read my posts at least once to avoid so many typos. Hopefully
>>>>> you are brave enough to read through.
>>>>>
>>>>> Petr
>>>>>
>>>>> On Mon, Sep 21, 2015 at 11:23 AM, Petr Novak <oss.mli...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I think you would have to persist events somehow if you don't want to
>>>>>> miss them. I don't see any other option there. Either in MQTT if it is
>>>>>> supported there or routing them through Kafka.
>>>>>>
>>>>>> There is WriteAheadLog in Spark but you would have decouple stream
>>>>>> MQTT reading and processing into 2 separate job so that you could upgrade
>>>>>> the processing one assuming the reading one would be stable (without
>>>>>> changes) across versions. But it is problematic because there is no easy
>>>>>> way how to share DStreams between jobs - you would have develop your own
>>>>>> facility for it.
>>>>>>
>>>>>> Alternatively the reading job could could save MQTT event in its the
>>>>>> most raw form into files - to limit need to change code - and then the
>>>>>> processing job would work on top of it using Spark streaming based on
>>>>>> files. I this is inefficient and can get quite complex if you would like 
>>>>>> to
>>>>>> make it reliable.
>>>>>>
>>>>>> Basically either MQTT supports prsistence (which I don't know) or
>>>>>> there is Kafka for these use case.
>>>>>>
>>>>>> Another option would be I think to place observable streams in
>>>>>> between MQTT and Spark streaming with bakcpressure as far as you could
>>>>>> perform upgrade till buffers fills up.
>>>>>>
>>>>>> I'm sorry that it is not thought out well from my side, it is just a
>>>>>> brainstorm but it might lead you somewhere.
>>>>>>
>>>>>> Regards,
>>>>>> Petr
>>>>>>
>>>>>> On Mon, Sep 21, 2015 at 10:09 AM, Jeetendra Gangele <
>>>>>> gangele...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have an spark streaming application with batch (10 ms) which is
>>>>>>> reading the MQTT channel and dumping the data from MQTT to HDFS.
>>>>>>>
>>>>>>> So suppose if I have to deploy new application jar(with changes in
>>>>>>> spark streaming application) what is the best way to deploy, currently 
>>>>>>> I am
>>>>>>> doing as below
>>>>>>>
>>>>>>> 1.killing the running streaming app using yarn application -kill ID
>>>>>>> 2. and then starting the application again
>>>>>>>
>>>>>>> Problem with above approach is since we are not persisting the
>>>>>>> events in MQTT we will miss the events for the period of deploy.
>>>>>>>
>>>>>>> how to handle this case?
>>>>>>>
>>>>>>> regards
>>>>>>> jeeetndra
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Reply via email to