Re: Beam SQL Improvements

Le 26 avr. 2018 23:13, "Anton Kedin" a écrit : BeamRecord (Row) has very little in common with JsonObject (I assume you're talking about javax.json), except maybe some similarities of the API. Few reasons why JsonObject doesn't work: - it is a Java EE API: - Beam SDK is not limited to J

Fwd: Apache Beam - jenkins question

Hi Kamil -- Thanks for reaching out. This is a great question for the dev@ mailing list. You may want to share a little bit more why you need, how long, frequency of updates to the secret, etc. for the community to be aware how things work. Hopefully others on the mailing list can help you by man

Re: Custom URNs and runner translation

On Thu, Apr 26, 2018 at 5:59 PM Eugene Kirpichov wrote: > I agree with Thomas' sentiment that cross-language IO is very important > because of how much work it takes to produce a mature connector > implementation in a language. Looking at implementations of BigQueryIO, > PubSubIO, KafkaIO, FileIO

Build failed in Jenkins: beam_Release_NightlySnapshot #743

See -- [...truncated 3.60 MB...] 2018-04-27T02:58:10.750 [INFO] Excluding io.dropwizard.metrics:metrics-json:jar:3.1.5 from the shaded jar. 2018-04-27T02:58:10.750 [INFO] Exc

Re: Custom URNs and runner translation

I agree with Thomas' sentiment that cross-language IO is very important because of how much work it takes to produce a mature connector implementation in a language. Looking at implementations of BigQueryIO, PubSubIO, KafkaIO, FileIO in Java, only a very daring soul would be tempted to reimplement

Build failed in Jenkins: beam_Release_NightlySnapshot #742

See -- [...truncated 1.76 MB...] 2018-04-27T00:13:19.180 [INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ beam-runners-extensions-java-metrics --- 2018-04-27T00

Re: Apache Beam Jenkins Machines Upgrade

This sounds awesome, thanks to everybody involved! On Thu, Apr 26, 2018 at 4:28 PM Yifan Zou wrote: > Greetings, > > Most of you already know about upgrades on Jenkins machines this week. I > still want to share some details you might interested in. > > In April 24th, we had 16 new GCE instances

Apache Beam Jenkins Machines Upgrade

Greetings, Most of you already know about upgrades on Jenkins machines this week. I still want to share some details you might interested in. In April 24th, we had 16 new GCE instances (beam9 - 24) setup under gcloud project 'apache-beam-testing' for Beam Jenkins. The old machines are currently d

Build failed in Jenkins: beam_Release_NightlySnapshot #741

See -- [...truncated 2.36 MB...] 2018-04-26T22:53:31.233 [INFO] 2018-04-26T22:53:31.233 [INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ beam-runners-extension

Jenkins build is back to normal : beam_SeedJob #1585

See

Build failed in Jenkins: beam_SeedJob #1584

See -- GitHub pull request #5235 of commit 7045f3287fc27a48f6a35a05b36a048dacf332d6, no merge conflicts. Setting status of 7045f3287fc27a48f6a35a05b36a048dacf332d6 to PENDING with url https

Re: Beam parent SNAPSHOTs are not being built

I can try triggering a one-off maven snapshot release. On Thu, Apr 26, 2018 at 1:59 PM Eric Beach wrote: > Is it possible to trigger a manual task to update the snapshot artifacts > here in the interim until a decision has been made? It is blocking dev > toward a public release. > > On Thu, Apr

Re: Beam SQL Improvements

BeamRecord (Row) has very little in common with JsonObject (I assume you're talking about javax.json), except maybe some similarities of the API. Few reasons why JsonObject doesn't work: - it is a Java EE API: - Beam SDK is not limited to Java. There are probably similar APIs for ot

Re: Beam parent SNAPSHOTs are not being built

Is it possible to trigger a manual task to update the snapshot artifacts here in the interim until a decision has been made? It is blocking dev toward a public release. On Thu, Apr 26, 2018 at 2:51 AM Romain Manni-Bucau wrote: > > > 2018-04-26 7:54 GMT+02:00 Reuven Lax : > >> This seems more tha

Re: Splittable DoFN in Spark discussion

Yeah that's been the implied source of being able to be continuous, you union with a receiver which produce an infinite number of batches (the "never ending queue stream" but not actually a queuestream since they have some limitations but our own implementation there of). On Tue, Apr 24, 2018 at 1

Re: Beam SQL Improvements

Just to let it be clear and let me understand: how is BeamRecord different from a JsonObject which is an API without implementation (not event a json one OOTB)? Advantage of json *api* are indeed natural mapping (jsonb is based on jsonp so no new binding to reinvent) and simple serialization (json+

Re: Beam SQL Improvements

Exactly what JB said. We will write a generic conversion from Avro (or json) to Beam schemas, which will make them work transparently with SQL. The plan is also to migrate Anton's work so that POJOs works generically for any schema. Reuven On Thu, Apr 26, 2018 at 1:17 AM Jean-Baptiste Onofré wro

Re: slack @the-asf?

An ASF slack admin generated another link for me and I updated the s.apache.org/slack-invite to work again. I also requested that ASF slack admins do this periodically instead of someone doing it each time. Until this is automated, when this breaks just ask for an ASF admin on #general such as @s

Re: Maven build broken

Thanks Thomas for the fix, Agree 100% maven should still work until the decision to drop it off is taken. On Thu, Apr 26, 2018 at 5:51 PM Thomas Weise wrote: > Here is a PR to fix it: > https://github.com/apache/beam/pull/5232 > I think that until switch to gradle is complete, at least "mvn ve

Re: Maven build broken

Here is a PR to fix it: https://github.com/apache/beam/pull/5232 I think that until switch to gradle is complete, at least "mvn verify" should continue to run in Jenkins pre-commit. Thanks, Thomas On Thu, Apr 26, 2018 at 4:53 AM, Jean-Baptiste Onofré wrote: > I would propose to revert in ord

Re: Beam SQL Improvements

Yes, that's my understanding where the Schema work is heading towards. Generic Row+Schema are in core java SDK and potentially can be backed by Avro or JSON or something else as an implementation/configuration detail. At the moment though the only implementation we have relies on RowCoder. On Thu,

Re: DoFn thread constraint.

Sure, wasn't aware of the site github.   Thanks!   John   - Original message -From: Romain Manni-Bucau To: dev@beam.apache.orgCc:Subject: Re: DoFn thread constraint.Date: Thu, Apr 26, 2018 9:18 AM  Do you want to give it a try doing a PR on https://github.com/apache/beam-site? Better to fix

Re: [BEAM-4088] Test isolation differences in Gradle

+1, thks Etienne for the hard work and to have spent time on it even if gradle was making it more complicated, really appreciated. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github <

Re: [BEAM-4088] Test isolation differences in Gradle

Hi guys, After some investigations, it appears to be a timing problem in the test. If we bloc until the end of the threads in metricsExecutorService before the ThreadLeakTracker rule is exectuted, then the test passes in gradle (no leaks detected). I guess we were lucky with the other build runs

Re: DoFn thread constraint.

Do you want to give it a try doing a PR on https://github.com/apache/beam-site? Better to fix an issue by the person getting the issue in general in that area ;) Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: DoFn thread constraint.

Thank you for your clarification!   May I suggest that clarification also make its way into the doc? We had some internal disagreement on reading the current text. :-)   John   - Original message -From: Romain Manni-Bucau To: dev@beam.apache.orgCc:Subject: Re: DoFn thread constraint.Date: T

Re: DoFn thread constraint.

Hi John, The fact a runner caches a fn per thread is an internal implementation detail but a fn will only be activated by one thread max at a time (like stateless or any object pool). This means your fn can fail. Romain Manni-Bucau @rmannibucau | Blog

DoFn thread constraint.

I'm trying to understand how restrictive this sentence from 4.3.2 of the Programming Guide is for our runner:   Each instance of your function object is accessed by a single thread on a worker instance, unless you explicitly create your own threads.   Does this imply that a runner may not ever use

Re: Maven build broken

I would propose to revert in order to give us time to fix. As we still have the two build tools, they have to build the same way. Regards JB Le 26 avr. 2018 à 14:57, à 14:57, Colm O hEigeartaigh a écrit: >Hi, > >This commit breaks the maven build: > >commit 5d5c052e890fb8af4bb9cd1a152300542feb

Maven build broken

Hi, This commit breaks the maven build: commit 5d5c052e890fb8af4bb9cd1a152300542feba0a2 Author: Ben Sidhom Date: Mon Apr 16 17:03:21 2018 -0700 [BEAM-4071] Add Portable Runner Job API shim The changes to runners/reference/java/build.gradle were not also made to the pom: + shadow projec

Re: Beam SQL Improvements

For now we have a generic schema interface. Json-b can be an impl, avro could be another one. Regards JB Le 26 avr. 2018 à 12:08, à 12:08, Romain Manni-Bucau a écrit: >Hmm, > >avro has still the pitfalls to have an uncontrolled stack which brings >way >too much dependencies to be part of any A

Re: Beam SQL Improvements

Hmm, avro has still the pitfalls to have an uncontrolled stack which brings way too much dependencies to be part of any API, this is why I proposed a JSON-P based API (JsonObject) with a custom beam entry for some metadata (headers "à la Camel"). Romain Manni-Bucau @rmannibucau

Re: Beam SQL Improvements

Hi Ismael You mean directly in Beam SQL ? That will be part of schema support: generic record could be one of the payload with across schema. Regards JB Le 26 avr. 2018 à 11:39, à 11:39, "Ismaël Mejía" a écrit: >Hello Anton, > >Thanks for the descriptive email and the really useful work. Any

Re: Beam SQL Improvements

Hello Anton, Thanks for the descriptive email and the really useful work. Any plans to tackle PCollections of GenericRecord/IndexedRecords? it seems Avro is a natural fit for this approach too. Regards, Ismaël On Wed, Apr 25, 2018 at 9:04 PM, Anton Kedin wrote: > Hi, > > I want to highlight a c

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #20

See Changes: [relax] Add support for @Element injection in NewDoFn. [relax] Add @Timestamp annotation to NewDoFn [relax] Add support for OutputReceiver and MultiOutputReceiver. Currently we [r