Hi Navina,
*>> after it has exhausted all messages from the source, it will generate a
special SENTINEL envelope that will not have any key or message and only
contain a special offset. Is that a valid understanding?*
The table in the doc already specifies how the sentinel envelope is
constructed
Thank you Renato for the insightful feedback! :-)
I don't think we have a complete scoping and design for the "punctuated
streams" feature as of yet. However, one way to implement it would be to
leverage something similar to what this feature does. We could have a
special offset for a "punctuation
It means that the message is not delivered to the "*StreamTask
implementation*" ie. user code. Yes, we are creating a contract between
SystemConsumer and Samza - " the SystemConsumer implementation will
generate an IncomingMessageEnvelope by invoking buildEndOfStream()"
>> Ok. Perhaps I didn't fram
Thanks Jagadish! This is great!
Can you share some thoughts/opinions on how this feature relates on using
punctuations (at some point) in Samza? I mean do you think that using
punctuated streams could be seen as a generalization of this problem? And
if so, could this feature be used later on as a b
Thanks for reviewing and the comments. Please find my replies inline.
*"The offset in the partition that the message was received from. Ifthis is
the last message in the SSP, this field is set to END_OF_STREAM.Such a
message is not delivered to the actual StreamTask implementation."Does thi
Hi Jagadish,
Thanks for sharing the design with the community. I have a couple of
questions that were not very clear from the design document.
1. Under mechanism for indicating the end-of-stream to Samza, you mention
"The offset in the partition that the message was received from. If
this is the l
> On Aug 30, 2016, at 4:44 PM, Xinyu Liu wrote:
>
> It's very exciting that Samza is adding support of bounded input streams.
+1!
It's very exciting that Samza is adding support of bounded input streams.
Nice write-up of different scenarios and options. Look forward to having
this feature work with the upcoming HDFS consumer!
Thanks,
Xinyu
On Tue, Aug 30, 2016 at 12:09 PM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrot
Currently, Samza works with streaming input sources like Kafka topics. This
proposal will build an idea of 'end-of-stream' into Samza to support data
sources that are bounded - like HDFS files, snapshots on disk etc.
Proposal:
https://issues.apache.org/jira/secure/attachment/12825119/ProposalforE
