Re: Primitive Read not working with Flink portable runner

Hi, I'd like to move the discussion of this topic further. Because it seems that fixing the portable SDF is a larger work, I think there are two options:  a) extend the definition of model coders to include SDK coders of the language that implement the model (that would mean that the definit

Re: Performance tests dashboard not working

Hi, I was back to check something in the dashboards and they seem to be failing with an error "Templating init failed NetworkError when attempting to fetch resource." Can somebody with more access/knowledge about that infra help me check what is going on. Thanks On Wed, Apr 21, 2021 at 9:41 PM

Critical vulnerabilities

Hi dev@apache, I recently opened a ticket (https://issues.apache.org/jira/browse/BEAM-12679) for dealing with some critical vulnerabilities in dependencies flagged by Twistlock, used by Beam. Just wanted to check that this is the right way to flag an issue of this kind? Please let me know if th

Re: Primitive Read not working with Flink portable runner

On Thu, Jul 29, 2021 at 3:04 AM Jan Lukavský wrote: > > Hi, > > I'd like to move the discussion of this topic further. Because it seems that > fixing the portable SDF is a larger work, I think there are two options: +1 > a) extend the definition of model coders to include SDK coders of the >

Re: Command line to run DatstoreIO integration tests for java

I was hoping for the command line to run it. So that the test could be tweaked to inject an error, and ensure the error handling code works as expected On Wed, Jul 28, 2021 at 8:34 PM Ke Wu wrote: > Comment the PR with "Run SQL PostCommit” would trigger the post commit > integration tests for SQ

Re: Primitive Read not working with Flink portable runner

On 7/29/21 6:47 PM, Reuven Lax wrote: On Thu, Jul 29, 2021 at 9:40 AM Robert Bradshaw > wrote: On Sun, Jul 25, 2021 at 11:59 AM Jan Lukavský mailto:je...@seznam.cz>> wrote: > > I didn't want to say that Flink should not support SDF. I only do not s

Re: Primitive Read not working with Flink portable runner

On 7/29/21 6:45 PM, Robert Bradshaw wrote: On Thu, Jul 29, 2021 at 3:04 AM Jan Lukavský wrote: Hi, I'd like to move the discussion of this topic further. Because it seems that fixing the portable SDF is a larger work, I think there are two options: +1 a) extend the definition of model c

Synchronization of RestrictionTrackers

Hi, I have come across something that looks like a bug to me, but I'm not sure of that. If I understand it correctly, RestrictionTracker.trySplit() and RestrictionTracker.tryClaim() methods are necessarily called from different threads. That implies, that modifying some fields inside these me

P1 issues report (45)

This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See https://beam.apache.

Re: Beam Website Feedback

Thank you and I filed https://issues.apache.org/jira/browse/BEAM-12682 to track this. (/cc @David Huntsperger ) Related question to everyone, do we have a process for the "Beam Website Feedback" emails? I see these emails occasionally, and some of them are for trivial issues and addressed quickly

RE: Synchronization of RestrictionTrackers

In practice, restriction tracker methods are called via a RestrictionTrackerObserver, which synchronizes (https://github.com/apache/beam/blob/master/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java) -Original Message- From: Jan Lukavský

Re: Synchronization of RestrictionTrackers

Ah, and that is created in OutputAndTimeBoundedSplittableProcessElementInvoker. That was the missing part, thanks. On 7/29/21 8:17 PM, Matt Rudary wrote: In practice, restriction tracker methods are called via a RestrictionTrackerObserver, which synchronizes (https://github.com/apache/beam/b

Flaky test issue report (31)

This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to determin

Re: Dropping @Experimental from SqlTransform

Makes sense to me. On Thu, Jul 29, 2021 at 11:12 AM Andrew Pilloud wrote: > > Hi everyone, > > Beam's Calcite SqlTransform has been stable for the last few years. We've > only been adding new methods and I don't expect us to make any breaking > changes. I believe it is time we drop the @Experim

Re: Critical vulnerabilities

Thanks. Filing a JIRA is fine. I don't think we use either of these dependencies in such a way that this poses a danger in Beam, but we should probably upgrade them just to be safe (and avoid any questions, including surfaced by automated tools). Fortunately the version bumps don't look that big.

Re: Command line to run DatstoreIO integration tests for java

“Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1] In your case where you would like to run DataStoreReadWriteIT only, it can be simplified [2] to "./gradlew :sdks:java:extensions:sql:postCommit” Best, Ke [1] https://github.com/apache/beam/blob/master/.test-infra/jenk