Does mapWithState checkpoints the data ? When my application goes down and is restarted from checkpoint, will mapWithState need to recompute the previous batches data ?
Also, to use mapWithState I will need to upgrade my application as I am using version 1.4.0 and mapWithState isnt supported there. Is there any other work around ? Cheers!! Abhi On Fri, Feb 12, 2016 at 3:33 AM, Sebastian Piu <sebastian....@gmail.com> wrote: > Looks like mapWithState could help you? > On 11 Feb 2016 8:40 p.m., "Abhishek Anand" <abhis.anan...@gmail.com> > wrote: > >> Hi All, >> >> I have an use case like follows in my production environment where I am >> listening from kafka with slideInterval of 1 min and windowLength of 2 >> hours. >> >> I have a JavaPairDStream where for each key I am getting the same key but >> with different value,which might appear in the same batch or some next >> batch. >> >> When the key appears second time I need to update a field in value of >> previous key with a field in the later key. The keys for which the >> combination keys do not come should be rejected after 2 hours. >> >> At the end of each second I need to output the result to external >> database. >> >> For example : >> >> Suppose valueX is object of MyClass with fields int a, String b >> At t=1sec I am getting >> key0,value0(0,"prev0") >> key1,value1 (1, "prev1") >> key2,value2 (2,"prev2") >> key2,value3 (3, "next2") >> >> Output to database after 1 sec >> key2, newValue (2,"next2") >> >> At t=2 sec getting >> key3,value4(4,"prev3") >> key1,value5(5,"next1") >> >> Output to database after 2 sec >> key1,newValue(1,"next1") >> >> At t=3 sec >> key4,value6(6,"prev4") >> key3,value7(7,"next3") >> key5,value5(8,"prev5") >> key5,value5(9,"next5") >> key0,value0(10,"next0") >> >> Output to database after 3 sec >> key0,newValue(0,"next0") >> key3,newValue(4,"next3") >> key5,newValue(8,"next5") >> >> >> Please suggest how this can be achieved. >> >> >> Thanks a lot !!!! >> Abhi >> >> >>
