RE: Apache Beam with Hive

Hi Pom attached. -Original Message- From: [google.com] Tomo Suzuki Sent: Wednesday, February 12, 2020 4:56 PM To: user@beam.apache.org Subject: Re: Apache Beam with Hive HI Noam, It seems incompatible libraries are in your dependencies. Would you share pom.xml (or build.gradle) that pr

Elasticsearch use in Apache Beam

Hi everyone, The Apache Beam community is currently working on refactoring the current ElasticsearchIO (see the thread [1] on the dev mailing list). To determine which Elasticsearch versions to support, we do a survey among users of Apache Beam. Can you please tell us more about your use of th

Achieving MapState Equivalent in Dataflow Runner

Hi All I'm developing a beam pipeline to be run on dataflow. I see that MapState and SetState are both not supported by Dataflow runner yet. For my use case, I need to have a state per UserID that can hold multiple events, which I would like to organize by an eventID or a timestamp or a combinati

Re: Achieving MapState Equivalent in Dataflow Runner

If the map/set can fit in memory then you can use a value state containing a Java Map/Set. On Thu, Feb 13, 2020 at 5:05 AM Ziyad Muhammed wrote: > Hi All > > I'm developing a beam pipeline to be run on dataflow. > > I see that MapState and SetState are both not supported by Dataflow runner > yet

Problems with side inputs on dataflow

Hello everybody I am facing a problem with a pipeline that runs perfectly on directrunner, but when it comes to dataflow, it turns into a mess. It changes the element and the side input (access). The side input reads only a line with credentials. Any thoughts on how its done are more than welcom

Re: Problems with side inputs on dataflow

If Dataflow changes the access pattern, it should also apply a wrapper function around how the side input is accessed and all of this should be effectively opaque to your execution. Can you provide more details as to what the "mess" is? On Thu, Feb 13, 2020 at 10:48 AM André Rocha Silva < a.si...

Java's SpannerAccessor package-private

Hi Apache Beam Java users, In Google-cloud-spanner dependency upgrade [1], I'm making org.apache.beam.sdk.io.gcp.spanner.SpannerAccessor package-private, as I believe this class is not touched by Beam users. If you need the class in your workload, let me know. [1]: https://github.com/apache/beam/

Re: Problems with side inputs on dataflow

It is getting access as an element and vice versa, it switches both. So I am never sure whether I am receiving (user token) or (thing1 thing2). On Thu, Feb 13, 2020 at 4:17 PM Luke Cwik wrote: > If Dataflow changes the access pattern, it should also apply a wrapper > function around how the

Re: Problems with side inputs on dataflow

That seems like a bug if Dataflow is reordering your parameter since you are correct in thinking that the side input parameters always follow the element parameters. I would suggest asking Google Cloud customer support about this and provide a job id/simple reproduction pipeline. On Thu, Feb 13,

Beam SQL Nested Rows are flatten by Calcite

Hi, I am trying to Beam SQL. But something is wrong. I have nested row records. I read them as Pcollection and apply Select * query and compare with initial rows. Looks like nested rows are flatten by calcite. How do you have any idea how can I avoid this? I added a same testcase for my issue: S

GCS numShards doubt

hi folks, I have this in code *globalIndexJson.apply("GCSOutput", TextIO.write().to(fullGCSPath).withSuffix(".txt").withNumShards(500));* the same code is executed for 50GB, 3TB, 5TB of data. I want to know if changing numShards for larger datasize will write to GCS faster?