Build failed in Jenkins: beam_SeedJob #1722

See Changes: [kmj] Bugfix: Read BQ bytes processed from correct field. [apilloud] [BEAM-3983] Add utils for converting to BigQuery types [apilloud] [BEAM-3983][SQL] Add BigQuery table provider [apilloud] [BEAM-424

Re: Beam high level directions (was "Graal instead of docker?")

The runner core doesnt fully align on that or rephrased more accurately, it doesnt go as far as it could for me. Having to call it, is still an issue since it requires a runner update instead of getting the new feature for free. The next step sounds to be *one* runner where implementations plug the

Build failed in Jenkins: beam_SeedJob_Standalone #1012

See Changes: [kmj] Bugfix: Read BQ bytes processed from correct field. [apilloud] [BEAM-3983] Add utils for converting to BigQuery types [apilloud] [BEAM-3983][SQL] Add BigQuery table provider [apilloud

Re: com.google.api.services.clouddebugger.v2.CloudDebugger ???

Are you running using 2.4.0 or HEAD ? We upgraded google-api-client dependencies of HEAD to 1.23 last month: https://github.com/apache/beam/pull/5046/files If you are using HEAD make sure that you are not picking up clouddebugger 1.22 (or any other 1.22 dependency). Thanks, Cham On Wed, May 16,

Re: Beam high level directions (was "Graal instead of docker?")

This -> "I'd like that each time you think that you ask yourself "does it need?"." On Wed, May 16, 2018 at 4:53 PM Robert Bradshaw wrote: > Thanks for your email, Romain. It helps understand your goals and where > you're coming from. I'd also like to see a thinner core, and agree it's > benefici

Re: Beam high level directions (was "Graal instead of docker?")

Thanks for your email, Romain. It helps understand your goals and where you're coming from. I'd also like to see a thinner core, and agree it's beneficial to reduce dependencies where possible, especially when supporting the usecase where the pipeline is constructed in an environment other than an

Re: [SQL] Cross Join Operation

It sounds like a good way to proceed would be to replace the template predicates with something that induces a real join. Kenn On Tue, May 15, 2018 at 9:33 AM Andrew Pilloud wrote: > Calcite does not have the concept of a "CROSS JOIN". It shows up in the > plan as a LogicalJoin with condition=[

Re: Java sdk tests are broken

This one: commit 05fa730267c0f3cd4bcd6e56e371fc7eb3d27131 Merge: 2927993ed 0c81cc90d Author: Thomas Groh Date: Wed May 16 09:50:54 2018 -0700 Merge pull request #5377: Roll forward handler delegation On Wed, May 16, 2018 at 2:36 PM Lukasz Cwik wrote: > Last successful postcommit was f

Re: Java sdk tests are broken

Last successful postcommit was from about an hour ago at commit 1e698901336b3499ad48d1bd2110963e45f364b7: https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/426/ What commit are you at? On Wed, May 16, 2018 at 1:30 PM Boyuan Zhang wrote: > Hey, > I pulled the late

Re: Performance Testing Dashboard - which results should be official?

Commented on the JIRA. I think this topic isn't so much about runner-to-runner comparison but just getting organized. For me working on a particular runner or IO or DSL the results are very helpful for seeing trends over time. On Wed, May 16, 2018 at 7:05 AM Jean-Baptiste Onofré wrote: > Hi Luka

Re: Java sdk tests are broken

Hey, I pulled the latest changes and ran ./gradlew -PisRelease clean build --no-parallel. Got same failure tests here: https://scans.gradle.com/s/cnrxmj7wbxkna/tests. Boyuan On Tue, May 15, 2018 at 3:45 PM Lukasz Cwik wrote: > I believe Thomas Groh reverted the offending PR with > https://github

Re: Beam high level directions (was "Graal instead of docker?")

Hi Romain, This gives a clear view of your perspective. I also recommend you ask around to those who have been working on Beam and big data processing for a long time to learn more about their perspective. Your "Beam Analysis" is pretty accurate about what we've been trying to build. I would say

ElasticsearchIOTest failed during gradle build

Hey all, I'm working on debugging the process of release process and when running ./gradlew -PisRelease clean build, I got several tests failed. Here is one build scan: https://scans.gradle.com/s/t4ryx7y3jhdeo/console-log?task=:beam-sdks-java-io-elasticsearch-tests-5:test#L3. Any idea about why th

Re: Checkpointing and Restoring BoundedSource

Hi Shen, The only guarantee made by splitAtFraction is that the primary source + residual source = original source - there are no other guarantees. For checkpointing, you can use the following pattern: When you want to checkpoint, call splitAtFraction(getFractionConsumed() + epsilon) or something l

Checkpointing and Restoring BoundedSource

Hi, After recovering from a checkpoint, is it correct to use BoundedSource.BoundedReader#splitAtFraction(double) to resume a BoundedSource? My concern is that the doc says "the new range would contain *approximately* the given fraction of the amount of data in the current range." Does the word *ap

Re: JDBC support for Beam SQL

Among these options I would lean towards option 1. We already support a lot of infrastructure to call into Calcite for non-JDBC path, so adding some code to generate config does not seem like a big of a deal, especially if it will be a supported way at some point in Calcite. Pulling implementation

Re: JDBC support for Beam SQL

IIUC in #2 Beam SQL would live on the other side of a JDBC boundary from any use of it (including the BeamSQL transform). I'm a bit worried we'll have a problem plumbing all the info we need, either now or later, especially if we make funky extensions to support our version of SQL. Kenn On Wed, M

JDBC support for Beam SQL

I'm currently adding JDBC support to Beam SQL! Unfortunately Calcite has two distinct entry points, one for JDBC and one for everything else (see CALCITE-1525). Eventually that will change, but I'd like to avoid having two versions of Beam SQL until Calcite converges on a single path for parsing SQ

Re: com.google.api.services.clouddebugger.v2.CloudDebugger ???

The Dataflow worker relies on this dependency to be supplied as part of the users application. Are you sure the way you build/package your application hasn't changed in the past few days? Note that the DataflowRunner has specified "google-api-services-clouddebugger" as a dependency[1]. The other i

Re: Performance Testing Dashboard - which results should be official?

Hi Lukasz, Thanks, gonna comment in the Jira. Generally speaking, I'm not a big fan to compare a runner versus another, because there are bunch of parameters that can influence the results. Regards JB On 16/05/2018 15:54, Łukasz Gajowy wrote: Hi all, I created an issue which I believe is

Performance Testing Dashboard - which results should be official?

Hi all, I created an issue which I believe is interesting in terms of what should be included in the Performance Testing dashboard and what shouldn't. Speaking more generally, we have to settle which results should be treated as official ones. The issue description contains my idea of solving it,

com.google.api.services.clouddebugger.v2.CloudDebugger ???

Something just starting breaking on me in the last day or so. I have the trivial program: public static void main(String[] args) { PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create(); Pipeline pipeline = Pipeline.create(options); } If I call it with no arguments,

Beam high level directions (was "Graal instead of docker?")

Hi guys, Since it is not the first time we have a thread where we end up not understanding each other, I'd like to take this as an opportunity to clarify what i'm looking for, in a more formal way. This assumes our misunderstandings come from the fact I mainly tried to fix issues one by ones, inst