Hey Jeremiah, Here's a standard pattern that Samza users in the community have adopted.
Configure your job to consume from your input-topic that contains a stream of requests. In your process() call: - If you need to retry your request, store it for later processing in a RocksDB store - Configure a change-log for the store so that you get recoverability In your window() call: - Scan each entry in your store and re-attempt your remote request - If the remote request succeeds, delete the entry from your store Please let me know if you had further questions. On Tue, Jun 5, 2018 at 10:16 AM, Jeremiah Adams <jad...@helixeducation.com> wrote: > > I was hoping I could use Windowing to process a topic every n-seconds for > some lengthy retry logic. E.g. some tasks can be retried n-times every > m-seconds if they fail. Failed attempts are re-queued for n-attempts before > being considered a real failure. e.g. POSTing to an endpoint fails due to > outage - so retry in the event the network connectivity can be restored. > > My ideas so far: > 1. monitor two topics - one is attempt, two is reattempt. Process these > separately with distinct Tasks. Re-queue the re-attempts and process them > as the appropriate timespan expires. Wasteful concerning processing/network > resources in doing so. The task will spend a ton of time just requeuing > waiting for the appropriate timespan. > 2. upon process() check rocksDB for reattemtps. Reprocess if so, then > process the original IncommingMessageEnvolope. Feels contrived - i'm > processing reattempts as a side-effect of processing a normal message. > > Keep in mind our data is nowhere near 'big'. > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > ________________________________________ > From: Jagadish Venkatraman <jagadish1...@gmail.com> > Sent: Tuesday, June 5, 2018 10:22 AM > To: dev@samza.apache.org > Subject: Re: MessageCollector and WindowableTask > > Hey Jeremiah, > > >> Please only use that MessageCollector object for sending messages, and > don't use it outside of the call to window(). Why MessageCollector should > not be used outside the call to window() in a WindowableTask? > > The collector and coordinator instances are expected to be used within a > context of a single window call. All this means is that a window should not > make any assumptions that the same instance of the collector will be > provided across invocations. > > >> Does this mean I should not pass MessageCollector around as a method > argument to child objects responsible for processing the window()? > > As long as those objects/helper methods are scoped within a single window > call, it should be fine. > > Can you please provide some background on what you are trying to do? > > Thanks, > Jagadish > > > > On Tue, Jun 5, 2018 at 6:55 AM, Jeremiah Adams <jad...@helixeducation.com> > wrote: > > > From th?e docs: > > > > > > "If you need to send messages to output streams, you can use the > > MessageCollector<https://url.emailprotection.link/?abxQT9w4aKBPgxuwj- > FWE3xNwSOvLPAXEqUg1ZOXefiB62v9ush8qrBPEEYxuqiJ-TKVEuQiECD5ysLF4m1RuXA~~ > > javadocs/org/apache/samza/task/MessageCollector.html> object passed to > > the window() method. Please only use that MessageCollector object for > > sending messages, and don't use it outside of the call to window()." > > > > > > Why MessageCollector should not be used outside the call to window() in a > > WindowableTask? > > > > Does this mean I should not pass MessageCollector around as a method > > argument to child objects responsible for processing the window()? > > > > > > Jeremiah Adams > > Software Engineer > > https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfI > erU3ZwsA3Gv_C0~<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp- > qKE3Xn2gNiZ3dlqAeSDA~> > > Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu- > KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~> | Twitter< > https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zwnjt5JsoFoNU2KX2 > AeBPVE~ > > HelixEducation> | Facebook<https://url.emailprotection.link/?aAmyAO_ > nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV > -dMSER5wl8zDVRr3RsS0OmcF9A~~> | > > LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz- > TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~> > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
