Re: How to create a custom trigger?

2017-06-07 Thread Lukasz Cwik
Just to circle back, making the Runner TestStream compatible would allow for exercising existing tests for trigger functionality. The DirectRunner is the only one which currently does this but we hope all Runners will follow so that each Runner has a suite of tests they can rely on to verify trigge

Re: How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi Kenn, Thanks for the explanation. Yes, I will test the runner directly. Shen On Wed, Jun 7, 2017 at 12:35 PM, Kenneth Knowles wrote: > Hi Shen, > > There are four points (at least) to why we don't currently have custom > triggers: > > 1. The existing trigger state machine implementations us

Re: How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi Ben, This is only for testing purposes. The runner has its own policy to handle watermarks. For example, it may decide to drop some watermarks if updates are too frequent. I was thinking about converting the watermark policy into an application-visible behavior and use the PAssert to check the

Re: How to create a custom trigger?

2017-06-07 Thread Kenneth Knowles
Hi Shen, There are four points (at least) to why we don't currently have custom triggers: 1. The existing trigger state machine implementations use low-level and difficult APIs that are not available to users. 2. If we make triggers a UDF then we need to cross the Fn API. It makes a lot more port

Re: How to create a custom trigger?

2017-06-07 Thread Ben Chambers
There hasn't been a need for user defined triggers and we've found it is really hard to create triggers with proper behavior. Can you elaborate on why you are trying to use triggers to understand watermarks in this way? It's not clear how this trigger would be useful to that understanding. On Wed

Re: How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi Lukasz, Thanks again for the suggestion. Is there any reason for not allowing users create custom triggers? Shen On Wed, Jun 7, 2017 at 12:13 PM, Lukasz Cwik wrote: > You should really take a look at TestStream and have runners integrate with > it instead. > > There are already several test

Re: How to create a custom trigger?

2017-06-07 Thread Lukasz Cwik
You should really take a look at TestStream and have runners integrate with it instead. There are already several tests which validate TestStream compatible runners to make sure their trigger evaluations are correct. On Wed, Jun 7, 2017 at 10:10 AM, Shen Li wrote: > Hi Lukasz, > > Thanks for th

Re: How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi Lukasz, Thanks for the suggestion. I am trying to test how the runner generates watermarks. So I would like to have the trigger to fire on every watermark advancing event. Shen On Wed, Jun 7, 2017 at 10:49 AM, Lukasz Cwik wrote: > Look into the AfterPane#elementCountAtLeast trigger as it se

Re: How to create a custom trigger?

2017-06-07 Thread Lukasz Cwik
Look into the AfterPane#elementCountAtLeast trigger as it seems to be the closest to your description. It fires as soon as any data is available. Are you sure you don't want some kind of watermark based trigger with just with a small interval size? On Wed, Jun 7, 2017 at 8:22 AM, Shen Li wrot

Re: How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi Lukasz, Thanks for your response. Is it possible to implement the following logic using existing triggers: always fire the trigger on a GlobalWindow whenever watermark advances? Shen On Wed, Jun 7, 2017 at 10:05 AM, Lukasz Cwik wrote: > Users are unable to create custom trigger implementati

Re: How to create a custom trigger?

2017-06-07 Thread Lukasz Cwik
Users are unable to create custom trigger implementations. If you tell us what you want your trigger to do, we may be able to suggest an alternative. On Wed, Jun 7, 2017 at 7:41 AM, Shen Li wrote: > Hi, > > I created a custom trigger class (XYZ) by extending the OnceTrigger. During > execution,

Re: Beam 2.0 Release Q and A

2017-06-07 Thread Jesse Anderson
One last call to fill out the Q and A. I'm going to publish it on Wednesday June 14. Please fill it out before Tuesday June 13. One thing to note, you don't have to be a committer or developer to add something to the doc. If you're using Beam in production, that will help others decide to use Beam

How to create a custom trigger?

2017-06-07 Thread Shen Li
Hi, I created a custom trigger class (XYZ) by extending the OnceTrigger. During execution, I got this error "Cannot translate trigger class XYZ to a runner-API proto." It seems that the Triggers.ProtoConverter class needs to declare a convertSpecific method for my trigger XYZ. How can I use my cus