I was hoping I could use Windowing to process a topic every n-seconds for some lengthy retry logic. E.g. some tasks can be retried n-times every m-seconds if they fail. Failed attempts are re-queued for n-attempts before being considered a real failure. e.g. POSTing to an endpoint fails due to outage - so retry in the event the network connectivity can be restored.
My ideas so far: 1. monitor two topics - one is attempt, two is reattempt. Process these separately with distinct Tasks. Re-queue the re-attempts and process them as the appropriate timespan expires. Wasteful concerning processing/network resources in doing so. The task will spend a ton of time just requeuing waiting for the appropriate timespan. 2. upon process() check rocksDB for reattemtps. Reprocess if so, then process the original IncommingMessageEnvolope. Feels contrived - i'm processing reattempts as a side-effect of processing a normal message. Keep in mind our data is nowhere near 'big'. Jeremiah Adams Software Engineer www.helixeducation.com Blog | Twitter | Facebook | LinkedIn ________________________________________ From: Jagadish Venkatraman <jagadish1...@gmail.com> Sent: Tuesday, June 5, 2018 10:22 AM To: dev@samza.apache.org Subject: Re: MessageCollector and WindowableTask Hey Jeremiah, >> Please only use that MessageCollector object for sending messages, and don't use it outside of the call to window(). Why MessageCollector should not be used outside the call to window() in a WindowableTask? The collector and coordinator instances are expected to be used within a context of a single window call. All this means is that a window should not make any assumptions that the same instance of the collector will be provided across invocations. >> Does this mean I should not pass MessageCollector around as a method argument to child objects responsible for processing the window()? As long as those objects/helper methods are scoped within a single window call, it should be fine. Can you please provide some background on what you are trying to do? Thanks, Jagadish On Tue, Jun 5, 2018 at 6:55 AM, Jeremiah Adams <jad...@helixeducation.com> wrote: > From th?e docs: > > > "If you need to send messages to output streams, you can use the > MessageCollector<https://url.emailprotection.link/?abxQT9w4aKBPgxuwj-FWE3xNwSOvLPAXEqUg1ZOXefiB62v9ush8qrBPEEYxuqiJ-TKVEuQiECD5ysLF4m1RuXA~~ > javadocs/org/apache/samza/task/MessageCollector.html> object passed to > the window() method. Please only use that MessageCollector object for > sending messages, and don't use it outside of the call to window()." > > > Why MessageCollector should not be used outside the call to window() in a > WindowableTask? > > Does this mean I should not pass MessageCollector around as a method > argument to child objects responsible for processing the window()? > > > Jeremiah Adams > Software Engineer > https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~> > Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~> > | > Twitter<https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zwnjt5JsoFoNU2KX2AeBPVE~ > HelixEducation> | > Facebook<https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~> > | > LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~> > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
