Adding new Jira component for Twister2

2020-08-27 Thread Pulasthi Supun Wickramasinghe
Hi All, While creating a issue for Twister2 i noticed currently there is no component tag for the Twister2 runner. Should we add a new component "runner-twister2", if so what are the steps to creating a component? Best Regards, Pulasthi -- Pulasthi S. Wickramasinghe PhD Candidate | Research As

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

2020-08-27 Thread Luke Cwik
As an update. Direct and Twister2 are done. Samza: is ready for review[1]. Flink: is almost ready for review. [2] lays all the groundwork for the migration and [3] finishes the migration (there is a timeout happening in FlinkSubmissionTest that I'm trying to figure out). No further updates on Spar

Re: [PROPOSAL] Preparing for Beam 2.24.0 release

2020-08-27 Thread Eugene Kirpichov
Hi Daniel, This is super helpful, thank you for the update! On Thu, Aug 27, 2020 at 2:09 PM Daniel Oliveira wrote: > Hey Eugene, > > That Jira is a bit misleading, it's still tracking a root cause, but a > workaround was submitted so it's no longer blocking the release. I'll > remove the releas

Re: [PROPOSAL] Preparing for Beam 2.24.0 release

2020-08-27 Thread Daniel Oliveira
Hey Eugene, That Jira is a bit misleading, it's still tracking a root cause, but a workaround was submitted so it's no longer blocking the release. I'll remove the release tag from it to avoid that confusion. I've been trying to get a release candidate out since last Thursday, but between several

Re: [PROPOSAL] Preparing for Beam 2.24.0 release

2020-08-27 Thread Eugene Kirpichov
Hi! Just wondering how the progress on 2.24 has been? I see the version in JIRA https://issues.apache.org/jira/projects/BEAM/versions/12347146 is blocked only by https://issues.apache.org/jira/browse/BEAM-10663 which hasn't seen much action in the last week. Is there something specific people can

Re: [External] Re: Memory Issue When Running Beam On Flink

2020-08-27 Thread Jan Lukavský
> If the user chooses to create a window of 10 years, I'd say it is expected behavior that the state will be kept for as long as this duration. State will be kept, the problem is that each key in the window will carry a cleanup timer, although there might be nothing to clear (there is no state

Re: [External] Re: Memory Issue When Running Beam On Flink

2020-08-27 Thread Maximilian Michels
If the user chooses to create a window of 10 years, I'd say it is expected behavior that the state will be kept for as long as this duration. GlobalWindows are different because they represent the default case where the user does not even use windowing. I think it warrants to be treated differ