Cool, thanks! Sorry for the late reply, we'll check out the UT and your design doc ASAP when we back from National Day holiday.
Thanks, Yuanjian Li Jungtaek Lim <kabh...@gmail.com> 于2018年9月29日周六 上午5:21写道： > Btw, just wrote up detailed design doc on existing patch: > > https://docs.google.com/document/d/1tUO29BDXb9127RiivUS7Hv324dC0YHuokYvyQRpurDY/edit?usp=sharing > > This doc is a wall of text, since I guess we already imagine how session > window works (and I showed a simple example in SPIP doc), so try to avoid > drawing something which would take non-trivial efforts. New classes are > linked to the actual source code so that we can read the code directly > whenever curious/wonders about something. > > Please let me know anytime if something is unclear and need elaboration. > > -Jungtaek Lim (HeartSaVioR) > > 2018년 9월 28일 (금) 오후 10:18, Jungtaek Lim <kabh...@gmail.com>님이 작성: > >> Thanks for sharing your proposal as well as implementation. Looks like >> your proposal is more like focused to design details: I may be better to >> write one more for design details and share it as well. Stay tuned! >> >> Btw, I'm trying out your patch to see whether it passes the tests I've >> added, and looks like it fails on below UT: >> >> https://github.com/apache/spark/blob/ad0b7466ef3f79354a99bd1b95c23e4c308502d5/sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala#L475-L573 >> Could you take a look at UT and see whether I'm missing here or the UT is >> correct? >> >> (Actually most of UTs I've added fail but some UTs are for update mode, >> and the patch doesn't provide same experience with select only session >> window, so I'm pointing only one UT which is testing basic session window.) >> >> -Jungtaek Lim (HeartSaVioR) >> >> 2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <xyliyuanj...@gmail.com>님이 작성: >> >>> Hi Jungtaek: >>> >>> We also meet this problem during migration of streaming application >>> to Structure Streaming in Baidu practice, we solved this in our folk and >>> just steady running in product. >>> As the initial plan we are doing the code clean work and preparing >>> give a SPIP in Oct, happy to see your proposal. Hope we share some spots >>> together. >>> Here’s the PR and doc: >>> https://github.com/apache/spark/pull/22583 >>> >>> https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing >>> >>> Thanks, >>> Yuanjian Li >>> >>> >>> 在 2018年9月28日，06:22，Jungtaek Lim <kabh...@gmail.com> 写道： >>> >>> Hi all, >>> >>> I would like to initiate discussion thread to discuss "Native support of >>> session window". >>> Origin issue is filed to SPARK-10816 [1] but I can file another one for >>> representing SPIP if necessary. WIP but working PR is available as well, so >>> we can even test it directly or see the difference if some of us feel more >>> convenient to go through the source code instead of doc. >>> >>> I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs >>> link [2] for who feel convenient to comment in doc. >>> >>> Please let me know if we would like to see also technical design for >>> this. I avoid to go too deep on SPIP doc so anyone could review and see the >>> benefit of adopting this. >>> >>> Looking forward to hear your feedback. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> 1. https://issues.apache.org/jira/browse/SPARK-10816 >>> 2. >>> https://docs.google.com/document/d/1_rMLmUSyGzb62RnP2A3WX6D6uRxox8Q_7WcoI_HrTw4/edit?usp=sharing >>> 3. https://github.com/apache/spark/pull/22482 >>> >>> >>> >>>
