[no subject]

Beam 2.13, I'm trying to do SqlTransform on Row records with hierarchical structures. Records having (nullable) arrays of (nullabke) sub-records: "select p.recordType, p.listOfServiceData.seqOf[0].ratingGroup as ratingGroup, p.abcdAddress.seqOf[0] as ABCD_IP FROM PCOLLECTION " The code resposibl

Re:

Hi Alex, Unfortunately you are receiving an UnsupportedOperationException because selecting nested rows is not supported by Calcite. You select fields out of the nested row but not a row itself. There are some recent bug fixes in this area in Calcite 1.20, so it might be worth trying that. There h

Beam vs Serverless (Dataflow vs Functions)

I need to take data that is sent over PubSub and simply store it into MongoDB. Is there an advantage to use Dataflow Streaming rather than Cloud Functions

Re: Beam vs Serverless (Dataflow vs Functions)

Its likely just a difference in cost and ease of implementation but Cloud Functions seems like it may fit your use case fine. On Thu, Jun 20, 2019 at 5:48 AM Joshua Fox wrote: > I need to take data that is sent over PubSub and simply store it into > MongoDB. > > Is there an advantage to use Dat

Re:

I wrote some tests on nested row selection in BeamSQL[1]. Those test cases test some behaviors of nested row selection that BeamSQL supports(but it's not a complete list). You could check what are tested so that are supported. Also it's welcome to extend those tests to cover more behaviors. [1]:

Re:

Oops I made a mistake, I didn't work on[1] but actually [2] [1]: https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java [2]: https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java

Re: Beam vs Serverless (Dataflow vs Functions)

In my case, I have the same scenario in production and I am using the *Cloud Function*, and it is working fine now. Even I am doing some lightweight processing for the data, and it is so far so good. Dataflow streaming is an alternative of course, but I didn't need it so far. On Fri, Jun 21, 2019