Re: Stream Progress

2016-01-21 Thread Yi Pan
Hi, Zach, We try to accommodate the low-watermark concept w/o depending on the embedded low-watermark messages in the streams. The dependency on low-watermark messages in the streams is a concern for us due to the following two reasons: 1) The injection of low-watermark messages depends on the pro

Re: Stream Progress

2016-01-19 Thread Zach Cox
Hi Yi, I read through the SAMZA-552 design [1] and have some questions: is the low-watermark concept included in the Window Metadata? Does the window operator compute the low-watermark from timestamps on incoming events? Thanks, Zach [1] https://issues.apache.org/jira/secure/attachment/12708934/

Re: Stream Progress

2016-01-12 Thread Yi Pan
Hi, Zach, Glad that you pointed it out! Actually, the design description in SAMZA-552 has adopt a lot of flavors of high-watermark, late-arrivals, from MillWheel. The terms used in the design doc maybe different since the terms in the doc were used earlier than we discovered the MillWheel presenta