Jenkins build is back to stable : beam_Release_NightlySnapshot #332

See

Performance Testing Next Steps

Hi all, I've written up a doc on next steps for getting performance testing up and running for Beam. I'd love to hear from people -- there's a fair amount of work encapsulated in here, but the end result is that we have a performance testing system which we can use for benchmarking all aspects of

Re: Metrics for Beam IOs.

@JB I think what you're suggesting is that Beam should provide a "Metrics Reporting" API as well, and I used to think like you, but the more I thought of that the more I tend to disagree now. The SDK is for users to author pipelines, so Metrics are for user-defined metrics (in contrast to runner m

Re: Metrics for Beam IOs.

It don't think it is possible for there to be a general mechanism for pushing metrics out during the execution of a pipeline. The Metrics API suggests that metrics should be reported as values across all attempts and values across only successful attempts. The latter requires runner involvement to

Re: Jenkins build is unstable: beam_Release_NightlySnapshot #331

Looking for ~5 seconds, failure looks like infrastructure related to ElasticSearchIO. I couldn't find a known issue or source of flakiness for ElasticSearchIO in JIRA, and I didn't have time to investigate whether it is a one-time issue or something actionable. Thoughts? On Thu, Feb 16, 2017 at 11

RE: Merge HadoopInputFormatIO and HDFSIO in a single module

Thank you all for your inputs! -Original Message- From: Dan Halperin [mailto:dhalp...@google.com.INVALID] Sent: Friday, February 17, 2017 12:17 PM To: dev@beam.apache.org Subject: Re: Merge HadoopInputFormatIO and HDFSIO in a single module Raghu, Amit -- +1 to your expertise :) On Th

Re: Pipeline Surgery and an interception-free future

That depends on how you determine which view you're trying to use. All of the nodes that are visited in Pipeline#traverseTopologically are wired up in such a way that their outputs correspond one-to-one with the pipeline independent user graph. However, that means that the contents of any override

Re: Metrics for Beam IOs.

Hi guys, As I'm back from vacation, I'm back on this topic ;) It's a great discussion, and I think about the Metric IO coverage, it's good. However, there's a point that we discussed very fast in the thread and I think it's an important one (maybe more important than the provided metrics ac

Re: Pipeline Surgery and an interception-free future

Thomas, how were you planning to get the StreamingViewAs* PTransforms out of the apply() in the Dataflow runner? I'm asking because the Flink Runner works basically the same way and I've run into a problem. My problem is that the TupleTag of a PCollectionView is generated when the user applies, fo